Who is Who and What is What: Experiments in Cross-Document Co-Reference

Proceedings of the 2008 Conference on Empirical Methods in Natural Language Processing, pages 274–283, Honolulu, October 2008. c⃝2008 Association for Computational Linguistics Who is Who and What is What: Experiments in Cross-Document Co-Reference Alex Baron BBN Technologies 10 Moulton Street Cambridge, MA 02138 abaron@bbn.com Marjorie Freedman BBN Technologies 10 Moulton Street Cambridge, MA 02138 mfreedma@bbn.com Abstract This paper describes a language-independent, scalable system for both challenges of cross- document co-reference: name variation and entity disambiguation. We provide system re- sults from the ACE 2008 evaluation in both English and Arabic. Our English system’s ac- curacy is 8.4% relative better than an exact match baseline (and 14.2% relative better over entities mentioned in more than one docu- ment). Unlike previous evaluations, ACE 2008 evaluated both name variation and entity disambiguation over naturally occurring named mentions. An information extraction engine finds document entities in text. We de- scribe how our architecture designed for the 10K document ACE task is scalable to an even larger corpus. Our cross-document ap- proach uses the names of entities to find an initial set of document entities that could refer to the same real world entity and then uses an agglomerative clustering algorithm to disam- biguate the potentially co-referent document entities. We analyze how different aspects of our system affect performance using ablation studies over the English evaluation set. In ad- dition to evaluating cross-document co- reference performance, we used the results of the cross-document system to improve the ac- curacy of within-document extraction, and measured the impact in the ACE 2008 within- document evaluation. 1 Introduction Cross-document entity co-reference is the problem of identifying whether mentions from different documents refer to the same or distinct entities. There are two principal challenges: the same entity can be referred to by more than one name string (e.g. Mahmoud Abbas and Abu Mazen) and the same name string can be shared by more than one entity (e.g. John Smith). Algorithms for solving the cross-document co-reference problem are neces- sary for systems that build knowledge bases from text, question answering systems, and watch list applications. There are several challenges in evaluating and developing systems for the cross-document co- reference task. (1) The annotation process required for evaluation and for training is expensive; an an- notator must cluster a large number of entities across a large number of documents. The annotator must read the context around each instance of an entity to make reliable judgments. (2) On randomly selected text, a baseline of exact string match will do quite well, making it difficult to evaluate pro- gress. (3) For a machine, there can easily be a scal- ability challenge since the system must cluster a large number of entities. Because of the annotation challenges, many previous studies in cross-document co-reference have focused on only the entity disambiguation problem (where one can use string retrieval to col- lect many documents that contain same name); or have used artificially ambiguated data. Section 2 describes related work; section 3 in- troduces ACE, where the …

Content truncated to 500 words