What You See is What You Get: Visual Pronoun Coreference Resolution in Dialogues

Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing, pages 5123–5132, Hong Kong, China, November 3–7, 2019. c⃝2019 Association for Computational Linguistics 5123 What You See is What You Get: Visual Pronoun Coreference Resolution in Dialogues Xintong Yu♥∗, Hongming Zhang♣∗, Yangqiu Song♣, Yan Song♠, and Changshui Zhang♥ ♥Institute for Artificial Intelligence, Tsinghua University (THUAI), Department of Automation, Tsinghua University ♣Department of CSE, The Hong Kong University of Science and Technology ♠Sinovation Ventures yuxt16@mails.tsinghua.edu.cn, hzhangal@cse.ust.hk, yqsong@cse.ust.hk, clksong@gmail.com, zcs@mail.tsinghua.edu.cn Abstract Grounding a pronoun to a visual object it refers to requires complex reasoning from var- ious information sources, especially in conver- sational scenarios. For example, when peo- ple in a conversation talk about something all speakers can see, they often directly use pro- nouns (e.g., it) to refer to it without previous introduction. This fact brings a huge chal- lenge for modern natural language understand- ing systems, particularly conventional context- based pronoun coreference models. To tackle this challenge, in this paper, we formally de- fine the task of visual-aware pronoun corefer- ence resolution (PCR) and introduce VisPro, a large-scale dialogue PCR dataset, to investi- gate whether and how the visual information can help resolve pronouns in dialogues. We then propose a novel visual-aware PCR model, VisCoref, for this task and conduct compre- hensive experiments and case studies on our dataset. Results demonstrate the importance of the visual information in this PCR case and show the effectiveness of the proposed model. 1 Introduction The question of how human beings resolve pro- nouns has long been an attractive research topic in both linguistics and natural language process- ing (NLP) communities, for the reason that pro- noun itself has weak semantic meaning (Ehrlich, 1981) and the correct resolution of pronouns re- quires complex reasoning over various informa- tion. As a core task of natural language under- standing, pronoun coreference resolution (PCR) (Hobbs, 1978) is the task of identifying the noun (phrase) that pronouns refer to. Compared with the general coreference resolution task, the string- matching methods are no longer effective for pro- nouns (Stoyanov et al., 2009), which makes PCR ∗ Equal contribution. A: What are they doing? B: Probably celebrating some holiday with such a big cake. A: Can you read the writing on it? B: I can’t tell. A: What is it behind the cake? the women the cake the statue Figure 1: An example of a visual-related dialogue. Two people are discussing the view they can both see. Pro- nouns and noun phrases referring to the same entity are marked in same color. The first “it” in the dialogue la- beled with blue color refers to the object “the big cake” and the second “it” labeled with green color refers to the statue in the image. more challenging than the general coreference res- olution task. Recently, great efforts have been devoted into the coreference resolution task (Raghunathan et al., 2010; Clark and Manning, 2015, 2016; Lee et al., 2018) …

Content truncated to 500 words