Cross-lingual Information Extraction System Evaluation

Cross-lingual Information Extraction System Evaluation Kiyoshi Sudo, Satoshi Sekine, and Ralph Grishman Department of Computer Science New York University 715 Broadway, 7th Floor, New York, NY 10003 � sudo,sekine,grishman � @cs.nyu.edu Abstract In this paper, we discuss the performance of cross- lingual information extraction systems employing an automatic pattern acquisition module. This mod- ule, which creates extraction patterns starting from a user’s narrative task description, allows rapid cus- tomization to new extraction tasks. We compare two approaches: (1) acquiring patterns in the source lan- guage, performing source language extraction, and then translating the resulting templates to the tar- get language, and (2) translating the texts and per- forming pattern discovery and extraction in the tar- get language. We demonstrate an average of 8-10% more recall using the first approach. We discuss some of the problems with machine translation and their effect on pattern discovery which lead to this difference in performance. 1 Introduction Research in information extraction (IE) and its re- lated fields has led to a wide range of applications in many domains. The portability issue of IE sys- tems across different domains, however, remains a serious challenge. This problem is being addressed through automatic knowledge acquisition methods, such as unsupervised learning for domain-specific lexicons (Lin et al., 2003) and extraction patterns (Yangarber, 2003), which require the user to pro- vide only a small set of lexical items of the target classes or extraction patterns for the target domain. The idea of a self-customizing IE system emerged recently with the improvement of pattern acquisi- tion techniques (Sudo et al., 2003b), where the IE system customizes itself across domains given by the user’s query. Furthermore, there are demands for access to in- formation in languages different from the user’s own. However, it is more challenging to provide an IE system where the target language (here, En- glish) is different from the source language (here, Japanese): a cross-lingual information extraction (CLIE) system. In this research, we explore various methods for efficient automatic pattern acquisition for the CLIE system, including the translation of the en- tire source document set into the target language. To achieve efficiency, the resulting CLIE system should (1) provide a reasonable level of extraction performance (both accuracy and coverage) and (2) require little or no knowledge on the user’s part of the source language. Today, there are basic linguis- tics tools available for many major languages. We show how we can take advantage of the tools avail- able for the source language to boost extraction per- formance. The rest of this paper is organized as follows. Section 2 and 3 discuss the self-adaptive CLIE sys- tem we assess throughout the paper. In Section 4, we show the experimental result for entity detec- tion. Section 5 discusses the problems in translation that affect the pattern acquisition and Section 6 dis- cusses related work. Finally, we conclude the paper in Section 7 with future work. 2 Query-Driven Information Extraction One approach to IE portability is to have a system that …

Content truncated to 500 words