Multi-feature Based Chinese-English Named Entity Extraction from Comparable Corpora

Multi-feature Based Chinese-English Named Entity Extraction from Comparable Corpora Min Lu, Jun Zhao 1 National Laboratory of Pattern Recognition, Institute of Automation, Chinese Academy of Sciences, Beijing, 100080, China { mlu, jzhao} @nlpr.ia.ac.cn Abstract. Bilingual Named Entity Extraction is important to some cross language information processes such as machine translation (MT), cross-lingual information retrieval (CLIR), etc. A lot of previous work extracted bilingual Named Entities from parallel corpus. Here we propose a multi- feature based method to extract bilingual Named Entities from comparable corpus. We first recognize the Chinese and English Named Entities respectively from the Chinese and English part of the comparable corpus. Then all the feature scores are calculated for every possible pair of Chinese and English Named Entities. At last we combine these feature scores together and decide which pairs are mutual translations. For translation score calculation, we didn’t use the formula of IBM model 1 like previous approach. In stead, we used a modified edit distance to take the order of words into consideration. Experiment shows that the F-score of this method increased by 11 %. And with the multi-feature integration strategy encouraging results are obtained. 1 Introduction Named Entities (NE) such as person names, location names, organization names, etc, carry essential information in human language [1]. Examples of some base Named Entities are as follows: Person name: 胡锦涛 Hu jintao 迈克尔·欧文 Michael Owen Location name: 渥太华 Ottawa 华盛顿 Washington Organization name: 教育部 Ministry of Education 国家安全委员会 National Security Council Bilingual Named Entity Extraction is very important to many cross language information processes such as machine translation (MT), cross-lingual information retrieval (CLIR), etc. In machine translation systems we translate the NEs in the sentences according to a bilingual NE list. Thus the list should be constantly updated to ensure correct translation of new NEs, since new NEs appear on the Internet everyday. This paper is about how to extract bilingual NEs from Comparable Corpora, which can be easily obtained from the Internet. A lot of work has been done to get bilingual NEs from parallel corpora, aligning English NEs and Chinese NEs in parallel sentences [2][3]. Although encouraging results have been obtained, the parallel corpus required in this approach is not easy to obtain. The construction of the parallel corpus costs huge human work and a long time. Compared with parallel corpus, comparable corpus can be obtained much more easily. Comparable corpus is one that contains non-sentence-aligned, non-translated bilingual documents that are topic- aligned. For example, newspaper articles from two sources in different languages, within the same window of published dates, can constitute a comparable corpus [4]. However, the bilingual NE extraction from comparable corpus is much more difficult than that from the parallel corpus. In parallel corpus, given a source NE to be translated in a source sentence, the translation candidates are only the NEs in the corresponding target sentence. But in comparable corpus the possible translations for a Chinese NE are NEs of the same type in the whole target language corpus. The search …

Content truncated to 500 words