A Statistical Model for Multilingual Entity Detection and Tracking

A Statistical Model for Multilingual Entity Detection and Tracking R. Florian, H. Hassan � , A. Ittycheriah, H. Jing N. Kambhatla, X. Luo, N. Nicolov, and S. Roukos I.B.M. T.J. Watson Research Center Yorktown Heights, NY 10598 {raduf,abei,hjing,nanda,xiaoluo, nicolas,roukos}@us.ibm.com � hanyh@eg.ibm.com Abstract Entity detection and tracking is a relatively new addition to the repertoire of natural language tasks. In this paper, we present a statistical language-independent framework for identify- ing and tracking named, nominal and pronom- inal references to entities within unrestricted text documents, and chaining them into clusters corresponding to each logical entity present in the text. Both the mention detection model and the novel entity tracking model can use arbitrary feature types, being able to integrate a wide array of lexical, syntactic and seman- tic features. In addition, the mention detec- tion model crucially uses feature streams de- rived from different named entity classifiers. The proposed framework is evaluated with sev- eral experiments run in Arabic, Chinese and English texts; a system based on the approach described here and submitted to the latest Au- tomatic Content Extraction (ACE) evaluation achieved top-tier results in all three evaluation languages. 1 Introduction Detecting entities, whether named, nominal or pronom- inal, in unrestricted text is a crucial step toward under- standing the text, as it identifies the important concep- tual objects in a discourse. It is also a necessary step for identifying the relations present in the text and populating a knowledge database. This task has applications in in- formation extraction and summarization, information re- trieval (one can get all hits for Washington/person and not the ones for Washington/state or Washington/city), data mining and question answering. The Entity Detection and Tracking task (EDT hence- forth) has close ties to the named entity recognition (NER) and coreference resolution tasks, which have been the focus of attention of much investigation in the recent past (Bikel et al., 1997; Borthwick et al., 1998; Mikheev et al., 1999; Miller et al., 1998; Aberdeen et al., 1995; Ng and Cardie, 2002; Soon et al., 2001), and have been at the center of several evaluations: MUC-6, MUC-7, CoNLL’02 and CoNLL’03 shared tasks. Usually, in com- putational linguistic literature, a named entity represents an instance of a name, either a location, a person, an or- ganization, and the NER task consists of identifying each individual occurrence of such an entity. We will instead adopt the nomenclature of the Automatic Content Extrac- tion program1 (NIST, 2003a): we will call the instances of textual references to objects or abstractions mentions, which can be either named (e.g. John Mayor), nominal (e.g. the president) or pronominal (e.g. she, it). An entity consists of all the mentions (of any level) which refer to one conceptual entity. For instance, in the sentence President John Smith said he has no comments. there are two mentions: John Smith and he (in the order of appearance, their levels are named and pronominal), but one entity, formed by the set {John Smith, he}. In this paper, we present a …

Content truncated to 500 words