The Curse of Performance Instability in Analysis Datasets: Consequences, Source, and Suggestions

Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing, pages 904–916, November 16–20, 2020. c⃝2020 Association for Computational Linguistics 904 Will I Sound Like Me? Improving Persona Consistency in Dialogues through Pragmatic Self-Consciousness Hyunwoo Kim Byeongchang Kim Gunhee Kim Department of Computer Science and Engineering Seoul National University, Seoul, Korea {hyunw.kim, byeongchang.kim}@vl.snu.ac.kr gunhee@snu.ac.kr https://vl.snu.ac.kr/projects/consistency Abstract We explore the task of improving persona con- sistency of dialogue agents. Recent models tackling consistency often train with additional Natural Language Inference (NLI) labels or attach trained extra modules to the genera- tive agent for maintaining consistency. How- ever, such additional labels and training can be demanding. Also, we find even the best- performing persona-based agents are insensi- tive to contradictory words. Inspired by social cognition and pragmatics, we endow existing dialogue agents with public self-consciousness on the fly through an imaginary listener. Our approach, based on the Rational Speech Acts framework (Frank and Goodman, 2012), can enforce dialogue agents to refrain from ut- tering contradiction. We further extend the framework by learning the distractor selection, which has been usually done manually or ran- domly. Results on Dialogue NLI (Welleck et al., 2019) and PersonaChat (Zhang et al., 2018) dataset show that our approach reduces contradiction and improves consistency of ex- isting dialogue models. Moreover, we show that it can be generalized to improve context- consistency beyond persona in dialogues. 1 Introduction In the study of dialogue agents, consistency has been a long-standing issue. To resolve this, much research has been conducted to endow dialogue agents with personas. Li et al. (2016) propose to encode persona in embeddings and Zhang et al. (2018) introduce a persona-conditioned dialogue dataset. On top of these works, many efforts have been made to improve consistency. In spite of such recent significant progress, there is much room for improving persona-based di- alogue agents. We observe that even the best performing persona-based generative models (See et al., 2019; Wolf et al., 2019b; Roller et al., 2020) I like to stay at home. Interlocutor Literal Agent: 𝑆! [Inconsistent] I like going outside. Interlocutor Self-Conscious Agent: 𝑆" [Consistent] I like going outside. I love Disneyland! I go there every week. ‘Will I sound like me?’ Figure 1: Illustration of the consistency issue in dia- logue. While a literal dialogue agent (S0) fails to de- liver a consistent persona, our self-conscious agent (S1) does so, by modeling an imaginary listener. Icons are designed by Nhor Phai and Vincent Le Moign. are highly insensitive to contradictory words, and thus fail to deliver consistent persona to the in- terlocutor (Figure 1). Also, extra modules other than the generative model is often required for im- proving consistency. Recent works on consistency in persona-based dialogue actively adopt the NLI- based approach (Welleck et al., 2019; Song et al., 2019; Li et al., 2020; Song et al., 2020), which have the following prerequisites. First, they require labeled pairs of persona sentences and dialogue ut- terances with three categories: entailment, neutral, and contradiction. Next, methods with …

Content truncated to 500 words