OpenEL: An Annotated Corpus for Entity Linking and Discourse in Open Domain Dialogue

Proceedings of the 13th Conference on Language Resources and Evaluation (LREC 2022), pages 2245–2256 Marseille, 20-25 June 2022 © European Language Resources Association (ELRA), licensed under CC-BY-NC-4.0 2245 OPENEL: An Annotated Corpus for Entity Linking and Discourse in Open Domain Dialogue Wen Cui1, Leanne Rolston2, Marilyn A. Walker1, Beth Ann Hockey2 1University of California, Santa Cruz {wcui7, mawalker}@ucsc.edu 2LivePerson Inc. {lrolston, bhockey}@liveperson.com Abstract Entity linking in dialogue is the task of mapping entity mentions in utterances to a target knowledge base. Prior work on entity linking has mainly focused on well-written articles such as Wikipedia, annotated newswire, or domain-specific datasets. We extend the study of entity linking to open domain dialogue by presenting the OPENEL corpus: an annotated multi-domain corpus for linking entities in natural conversation to Wikidata. Each dialogic utterance, in 179 dialogues over 12 topics from the original EDINA corpus, has been annotated for entities realized by definite referring expressions as well as anaphoric forms such as he, she, it and they. OPENEL thus supports training and evaluation of entity linking in open-domain dialogue, as well as analysis of the effect of using dialogue context and anaphora resolution in model training. It can also be used for fine-tuning a coreference resolution algorithm. To the best of our knowledge, this is the first substantial entity linking corpus publicly available for open-domain dialogue. We also establish baselines for named entity linking in open domain conversation using several existing entity linking systems. We find that the Transformer-based system, Flair + BLINK, has the best performance with a 0.65 F1 score. Our results show that dialogue context is extremely beneficial for entity linking in conversations, with Flair + BLINK achieving an F1 of 0.61 without discourse context. These results also demonstrate the remaining performance gap between the baselines and human performance, highlighting the challenges of entity linking in open-domain dialogue, and suggesting many avenues for future research using OPENEL. Keywords: Entity Linking, Coreference, Discourse Modeling, Wikidata, Open-domain Dialogue 1. Introduction Named entity recognition (NER), named entity link- ing (NEL) and discourse modeling (DM) are crucial aspects of natural language understanding (NLU) for open-domain dialogue systems, such as chatbots or so- cialbots. Unlike task-oriented dialogue systems, open- domain systems need to be able to converse on any topic, with the concomitant challenge of handling a much wider variety of named entities. NEL provides grounding of the named entities in a knowledge base (KB) (in this work Wikidata) by assigning a knowledge base ID to each named entity mention and anaphoric reference. This is demonstrated in the dialogue excerpt in Figure 1, where we see The Other Woman intro- duced by speaker A in A1, and then referred to in B1 as that. In A2, speaker A then refers to Cameron Diaz and Leslie Mann by the possessive referring expression their. The speakers continue, throughout the dialogue, to build on entities already added to the discourse model, as they comment further on entities introduced earlier in the dialogue, and introduce related entities, e.g. A3 …

Content truncated to 500 words