Generating Chinese Named Entity Data from a Parallel Corpus

Proceedings of the 5th International Joint Conference on Natural Language Processing, pages 264–272, Chiang Mai, Thailand, November 8 – 13, 2011. c⃝2011 AFNLP Generating Chinese Named Entity Data from a Parallel Corpus Ruiji Fu, Bing Qin, Ting Liu* Research Center for Social Computing and Information Retrieval MOE-Microsoft Key Laboratory of Natural Language Processing and Speech School of Computer Science and Technology Harbin Institute of Technology, Harbin, China {rjfu,bqin,tliu}@ir.hit.edu.cn Abstract Annotating Named Entity Recognition (NER) training corpora is a costly process but necessary for supervised NER systems. This paper presents an ap- proach to generate large-scale Chinese NER training data from an English- Chinese discourse level aligned parallel corpus. Difficulty of NER is different among languages due to their unique fea- tures. For example, the performance of English NER systems is usually higher than the Chinese ones on average. In our method, we first employ a high perfor- mance NER system on one side of a bi- lingual corpus. And then, we project the NE labels to the other side according to the word level alignment. At last, we se- lect high-quality labeled sentences using different strategies and generate an NER training corpus. In our experiments, we generate a Chinese NER corpus with 167,100 sentences through an English- Chinese parallel corpus. The system trained on the automatically generated corpus attains a comparable result with the one trained on the manually- annotated corpus. Further experiments show that the NER performance is signif- icantly improved on two different evalua- tion sets by using the generated training data as an additional corpus to the ma- nually-labeled data. 1 Introduction Named Entity Recognition (NER) is the task of identifying and classifying the names of persons, * Correspondence author: tliu@ir.hit.edu.cn locations, organizations and other named entities in text, which plays an important role in many Natural Language Processing (NLP) applications such as information extraction, information re- trieval, machine translation, and so on. Supervised machine learning systems have proved successful for NER (Zhou and Su, 2002; Chieu and Ng, 2002; Takeuchi and Collier, 2002; Settles, 2004). They usually need manually- annotated high performance textual corpora. These corpora are considered as gold standards for training statistical models. However, corpora manually-annotating is so costly and time- consuming that the existing corpora are limited in both scale and scope for Chinese NER. More seriously, the domain overfitting prob- lem even worsens the corpora-shortage problem. Supervised NER approaches can often achieve high accuracy when a large annotated training set similar to the test data is available (Zhou and Su, 2002; Florian et al., 2003; Klein et al., 2003; Finkel et al., 2005). Unfortunately, if the test data has some difference from the training data, these approaches tend to not perform well. For in- stance, Ciaramita and Altun (2005) reported that the F1-score of a named entity recognizer trained on CoNLL 2003 Reuters corpus dropped from 90.8% (when tested on a similar Reuters set) to 64.3% (when tested on a Wall Street Journal set). A similar phenomenon of performance degrada- tion in …

Content truncated to 500 words