What Did You Refer to? Evaluating Co-References in Dialogue

Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021, pages 5075–5084 August 1–6, 2021. ©2021 Association for Computational Linguistics 5075 What did you refer to? Evaluating Co-references in Dialogue Weinan Zhang♯, Yue Zhang♭, Hanlin Tang♦, Zhengyu Zhao♯, Caihai Zhu♯, Ting Liu♯ ♯Harbin Institute of Technology, Harbin, China ♭Westlake University, Hangzhou, China ♦School of Computer Science and Technology, Beijing, China {wnzhang, zyzhao, chzhu, tliu}@ir.hit.edu.cn zhangyue@westlake.edu.cn hltang@bit.edu.cn Abstract Existing neural end-to-end dialogue models have limitations on exactly interpreting the lin- guistic structures, such as ellipsis, anaphor and co-reference, etc., in dialogue history context. Therefore, it is hard to determine whether the dialogue models truly understand a dialogue or not, only depending on the coherence evalu- ation of their generated responses. To address these issues, in this paper, we proposed to di- rectly measure the capability of dialogue mod- els on understanding the entity-oriented struc- tures via question answering and construct a new benchmark dataset, DEQA, includ- ing large-scale English and Chinese human- human dialogues. Experiments carried on rep- resentative dialogue models show that these models all face challenges on the proposed di- alogue understanding task. The DEQA dataset will release for research use. 1 Introduction Driven by the growth of interest in social chat- bot, online customer service and virtual mobile assistant, social dialogue systems have received in- creasing research attention (Cui et al., 2017; Zhou et al., 2018; Hancock et al., 2019). The current dominant method has been sequence-to-sequence models, trained over large dialogue data end-to-end. Such models use neural network architectures such as Transformer (Vaswani et al., 2017) to encode a user utterance and a dialogue history before gener- ating a system utterance (Adiwardana et al., 2020; Roller et al., 2020; Bao et al., 2020). A major ad- vantage is the use of standard and general model architecture, which facilitates end-to-end training process over large scale dialogue text (Shang et al., 2015; Zhang et al., 2018, 2019, 2020). 1IDENT denotes the entities in a co-reference chain are identical. “1.6-8” indicates that “a clean house” is from the 6th to 8th tokens in the 1st utterance. Dialogue U1: Well, you know how important a clean house is to your grandma. U2: Yes, I hear about it every time she comes here. What do you hear about? Q1 A clean house. A1 U1: She was the head janitor at St. Mary’s Hos- pital for thirty years, after all. U2: I think she misses that job and wants to take it out on us. U1: You know, maybe she is just a neat freak. Who is just a neat freak? Q2 Grandma. A2 U2: I think she just likes to make us miserable. U1: You could be right. (a) Co-reference Chain (OntoNotes style) Chain 1 (IDENT) Chain 3 (IDENT) 1.6-8 a clean house 1.12-12 grandma 2.5-5 it 2.8-8 she Chain 2 (IDENT) 3.1-1 she 3.4-9 head janitor at St. 4.3-3 she Mary’s Hospital 5.4-4 she 4.5-6 that job 6.3-3 she (b) Table 1: (a) Sample of English dialogue in the proposed dataset. U1 …

Content truncated to 500 words