SemEval 2018 Task 4: Character Identification on Multiparty Dialogues

Proceedings of the 12th International Workshop on Semantic Evaluation (SemEval-2018), pages 57–64 New Orleans, Louisiana, June 5–6, 2018. ©2018 Association for Computational Linguistics SemEval 2018 Task 4: Character Identification on Multiparty Dialogues Jinho D. Choi Computer Science Emory University Atlanta, GA 30322 jinho.choi@emory.edu Henry Y. Chen Information Security Snap Inc. Santa Monica, CA 90405 henry.chen@snapchat.com Abstract Character identification is a task of entity link- ing that finds the global entity of each personal mention in multiparty dialogue. For this task, the first two seasons of the popular TV show Friends are annotated, comprising a total of 448 dialogues, 15,709 mentions, and 401 entities. The personal mentions are detected from nomi- nals referring to certain characters in the show, and the entities are collected from the list of all characters in those two seasons of the show. This task is challenging because it requires the identification of characters that are mentioned but may not be active during the conversation. Among 90+ participants, four of them submit- ted their system outputs and showed strengths in different aspects about the task. Thorough analyses of the distributed datasets, system out- puts, and comparative studies are also provided. To facilitate the momentum, we create an open- source project for this task and publicly release a larger and cleaner dataset, hoping to support researchers for more enhanced modeling. 1 Introduction Most of the earlier works in natural language pro- cessing (NLP) had focused on formal writing such as newswires, whereas many recent works have tar- geted at colloquial writing such as text messages or social media. Since the evolution of Web 2.0, the amount of user-generated contents involving col- loquial writing has exceeded the one with formal writing. NLP tasks are relatively well-explored at this point for certain types of colloquial writing i.e., microblogs and reviews (Ritter et al., 2011; Kong et al., 2014; Ranganath et al., 2016; Shin et al., 2017). However, the genre of multiparty dialogue is still under-explored, even though digital contents in dialogue forms keep increasing at a faster rate than any other types of writing.1 This inspires us 1https://medium.com/hijiffy/10-graphs-that-show-the- immense-power-of-messaging-apps-4a41385b24d6 to create a new task called character identification that aims to link personal mentions (e.g, she, mom) to their global entities across multiple dialogues, where the entities indicate the specific characters referred by those mentions (e.g., Judy). Due to the nature of multiparty dialogue where several speakers take turns to complete a context, character identification is a crucial step for adapting higher-end NLP tasks (e.g., summarization, ques- tion answering, machine translation) to this genre. It can also bring another level of sophistication to intelligent personal assistants or tutoring systems. This task is challenging because it needs to process through colloquialism that includes slangs, gram- mar mistakes, and/or rhetorical questions, as well as to handle cross-document resolution for the iden- tification of entities that are mentioned but may not be actively participating during the conversation. Nonetheless, we believe that models produced by this task will remarkably enhance inference on di- alogue contexts …

Content truncated to 500 words