Realistic Evaluation Principles for Cross-document Coreference Resolution

Proceedings of the 10th Conference on Lexical and Computational Semantics, pages 143–151 August 5–6, 2021, Bangkok, Thailand (online) ©2021 Association for Computational Linguistics 143 Realistic Evaluation Principles for Cross-document Coreference Resolution Arie Cattan1 Alon Eirew1,2 Gabriel Stanovsky3 Mandar Joshi4 Ido Dagan1 1Computer Science Department, Bar Ilan University 2Intel Labs, Israel 3The Hebrew University of Jerusalem 4Allen School of Computer Science & Engineering, University of Washington, Seattle, WA arie.cattan@gmail.com alon.eirew@intel.com gabis@cse.huji.ac.il mandar90@cs.washington.edu dagan@cs.biu.ac.il Abstract We point out that common evaluation prac- tices for cross-document coreference resolu- tion have been unrealistically permissive in their assumed settings, yielding inflated re- sults. We propose addressing this issue via two evaluation methodology principles. First, as in other tasks, models should be evaluated on predicted mentions rather than on gold men- tions. Doing this raises a subtle issue regard- ing singleton coreference clusters, which we address by decoupling the evaluation of men- tion detection from that of coreference linking. Second, we argue that models should not ex- ploit the synthetic topic structure of the stan- dard ECB+ dataset, forcing models to confront the lexical ambiguity challenge, as intended by the dataset creators. We demonstrate empiri- cally the drastic impact of our more realistic evaluation principles on a competitive model, yielding a score which is 33 F1 lower com- pared to evaluating by prior lenient practices.1 1 Introduction Cross-document (CD) coreference resolution iden- tifies and links textual mentions that refer to the same entity or event across multiple documents. For example, Table 1 depicts different news stories involving former U.S. president Barack Obama. While subsuming the challenges of within- document (WD) coreference, CD coreference in- troduces additional unique challenges. Most no- tably, lexical similarity is often not a good indi- cator when identifying cross-document links, as documents are authored independently. As shown in Table 1, the same event can be referenced using different expressions (“nominated”, “approached”), while two different events can be referenced us- ing the same expression (“name”). Despite these challenges, reported state-of-the-art results on the 1https://github.com/ariecattan/coref Subtopic 1 Doc 1: News that Barack Obama may name Dr. Sanjay Gupta of Emory University and CNN as his Surgeon... Doc 2: CNN’s management confirmed yesterday that Dr. Gupta had been approached by the Obama team. Subtopic 2 Doc 3: President Obama will name Dr. Regina Benjamin as Surgeon General in a Rose Garden announcement... Doc 4: Obama nominates new surgeon general: genius grant fellow Dr. Benjamin. He emphasizes his decision.. Table 1: Example of sentences of from the ECB+. The underlined words represent events, same color repre- sents a coreference cluster. Different documents de- scribe the same event using different words (e.g name, approached), while the two predicates “name” in the two subtopics are not coreferring. popular CD coreference ECB+ benchmark (Cybul- ska and Vossen, 2014) are relatively high, reaching up to 80 F1 (Barhom et al., 2019; Meged et al., 2020). In this paper, we show that CD coreference mod- els achieve these numbers using overly-permissive evaluation protocols, namely assuming gold entity and event mentions are …

Content truncated to 500 words