Contextual Clues for Named Entity Recognition in Japanese

©2002-2011 The CJK Dictionary Institute, Inc.

1. Names Entity Recognition (NER)

Identifying names and their variants is a difficult computational linguistic task. Named Entity Recognition (NER) is one of the hottest topics in computational linguistics, and plays a major role in NLP applications such as question answering, machine translation, information extraction, and the like. Much work has been done on developing NER tools based on statistical methods, but performing NER accurately is beyond the ability of such methods alone, which must be supplemented by large-scale name databases to be truly effective. However, the small-scale lexicons currently used are inadequate to the task.

2. Lexical Resources for NER

To meet this need, CJKI maintains comprehensive databases of several million proper nouns (personal names and place names), especially of Japanese names and Chinese names, as well as company and organization names. More details can be found at:

• Japanese names: japname.htm
• Chinese names: chinname.htm
• Korean names: korname.htm
• Arabic names: arabname.htm
• Spanish names

These resources are of major importance to NER tool developers, especially because they include millions of variants in all the major and most minor romanization systems, including the major Chinese dialects such as Cantonese, Hakka and Hokkien.

3. Contextual Clues for Japanese NER

Even large-scale proper noun databases cannot be kept fully up-to-date as new names are created daily. Moreover, many kinds of named entities are created arbitrarily and have only an ephemeral existence. Various techniques have been used to automatically detect named entities. Some of these issues are discussed in The Role of Lexical Resources in CJK NLP Applications.

A major technique is the use of keywords or syntactic structures that precede or follow named entities, which we refer to as named entity contextual clues (NECC). We have developed a comprehensive database of contextual clues for Japanese named entities that can play a critical role in enhancing the precision of NER applications.

The table below shows example of Japanese NECCs classified by type. Other attributes are available to help fine tune the data to specific NER requirements. In the example column, the red portion indicates the NECC.