Profile Consistency Identification for Open-domain Dialogue Agents

Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing, pages 6651–6662, November 16–20, 2020. c⃝2020 Association for Computational Linguistics 6651 Profile Consistency Identification for Open-domain Dialogue Agents Haoyu Song1, Yan Wang, Wei-Nan Zhang1, Zhengyu Zhao1, Ting Liu1, Xiaojiang Liu 1Research Center for Social Computing and Information Retrieval Harbin Institute of Technology, Heilongjiang, China {hysong,wnzhang,zyzhao,tliu}@ir.hit.edu.cn yanwang.branden@gmail.com xiaojiangliu84@hotmail.com Abstract Maintaining a consistent attribute profile is crucial for dialogue agents to naturally con- verse with humans. Existing studies on im- proving attribute consistency mainly explored how to incorporate attribute information in the responses, but few efforts have been made to identify the consistency relations between re- sponse and attribute profile. To facilitate the study of profile consistency identification, we create a large-scale human-annotated dataset with over 110K single-turn conversations and their key-value attribute profiles. Explicit rela- tion between response and profile is manually labeled. We also propose a key-value structure information enriched BERT model to identify the profile consistency, and it gained improve- ments over strong baselines. Further evalu- ations on downstream tasks demonstrate that the profile consistency identification model is conducive for improving dialogue consistency. 1 Introduction Despite the recent advancements in assigning at- tribute profiles to dialogue agents (Qian et al., 2018; Zhang et al., 2019), maintaining a consistent pro- file is still challenging for an open-domain dialogue agent. Existing works mainly emphasize the incor- poration of attribute information in the generated responses (Wolf et al., 2019; Song et al., 2019; Zheng et al., 2020). Although these models have improved the response consistency by explicitly modeling the profiles, they still face the consis- tency issue (Welleck et al., 2019). One important reason is that they cannot identify the consistency relations between response and profile. As shown in Figure 1, the attribute word Beijing is incorporated in the first two responses, but only R1 is semantically consistent with the speaker’s profile. For example, R2 “I also hope to visit Bei- jing one day.” implies that the speaker has never Gender Female Name Elena Current Location Beijing Constellation Aquarius Age Post-90s R1: I am glad you could come to Beijing. R3: I'll show you around Tsinghua University. R2: I also hope to visit Beijing one day. Query: I will go to Beijing tomorrow� Entailed Contradicted Figure 1: Left: the key-value attribute profiles of the dialogue agent. Right: a dialogue query with different responses that might be related to the attribute profiles. Among these responses, R1 entails the current location profile, while R2 contradicts the profile. Although R3 does not contain the attribute word Beijing, we could still understand R3 entails the current location. been to Beijing, which contradicts the speaker’s profile. On the other hand, although R3 does not contain the attribute word Beijing, we could still infer from the words Tsinghua University that the speaker’s current location entails the profile. Existing studies (Qian et al., 2018; Zheng et al., 2019) train dialogue agents to produce plausible re- sponses that contain attribute information, but still cannot teach agents to understand …

Content truncated to 500 words