Japanese Named Entity Recognition based on a Simple Rule Generator and Decision Tree Learning

Japanese Named Entity Recognition based on a Simple Rule Generator and Decision Tree Learning Hideki Isozaki NTT Communication Science Laboratories 2-4 Hikaridai, Seika-cho, Souraku-gun, Kyoto 619-0237, Japan isozaki@cslab.kecl.ntt.co.jp Abstract Named entity (NE) recognition is a task in which proper nouns and nu- merical information in a document are detected and classified into categories such as person, organization, location, and date. NE recognition plays an es- sential role in information extraction systems and question answering sys- tems. It is well known that hand-crafted systems with a large set of heuris- tic rules are difficult to maintain, and corpus-based statistical approaches are expected to be more robust and require less human intervention. Several statis- tical approaches have been reported in the literature. In a recent Japanese NE workshop, a maximum entropy (ME) system outperformed decision tree sys- tems and most hand-crafted systems. Here, we propose an alternative method based on a simple rule generator and decision tree learning. Our exper- iments show that its performance is comparable to the ME approach. We also found that it can be trained more efficiently with a large set of training data and that it improves readability. 1 Introduction Named entity (NE) recognition is a task in which proper nouns and numerical informa- tion in a document are detected and classi- fied into categories such as person, organiza- tion, location, and date. NE recognition plays an essential role in information extraction sys- tems (see MUC documents (1996)) and ques- tion answering systems (see TREC-QA docu- ments, http://trec.nist.gov/). When you want to know the location of the Taj Ma- hal, traditional IR techniques direct you to rele- vant documents but do not directly answer your question. NE recognition is essential for finding possible answers from documents. Although it is easy to build an NE recognition system with mediocre performance, it is difficult to make it re- liable because of the large number of ambiguous cases. For instance, we cannot determine whether “Washington” is a person’s name or a location’s name without the necessary context. There are two major approaches to building NE recognition systems. The first approach employs hand-crafted rules. It is well known that hand- crafted systems are difficult to maintain because it is not easy to predict the effect of a small change in a rule. The second approach employs a statis- tical method, which is expected to be more robust and to require less human intervention. Several statistical methods have been reported in the liter- ature (Bikel et al., 1999; Borthwick, 1999; Sekine et al., 1998; Sassano and Utsuro, 2000). IREX (Information Retrieval and Extraction Exercise, (Sekine and Eriguchi, 2000; IRE, 1999)) was held in 1999, and fifteen systems par- ticipated in the formal run of the Japanese NE ex- cercise. In the formal run, participants were re- quested to tag two data sets (GENERAL and AR- REST), and their scores were compared in terms of F-measure, i.e., the harmonic mean of ‘recall’ and ‘precision’ defined as follows. � recall = x/(the number of …

Content truncated to 500 words