I like fish, especially dolphins: Addressing Contradictions in Dialogue Modeling

Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing, pages 1699–1713 August 1–6, 2021. ©2021 Association for Computational Linguistics 1699 I like fish , especially dolphins :∗ Addressing Contradictions in Dialogue Modeling Yixin Nie1, Mary Williamson2, Mohit Bansal1, Douwe Kiela2, Jason Weston2 1UNC Chapel Hill 2Facebook AI Research Abstract To quantify how well natural language un- derstanding models can capture consistency in a general conversation, we introduce the DialoguE COntradiction DEtection task (DE- CODE) and a new conversational dataset con- taining both human-human and human-bot contradictory dialogues. We show that: (i) our newly collected dataset is notably more effec- tive at providing supervision for the dialogue contradiction detection task than existing NLI data including those aimed to cover the dia- logue domain; (ii) Transformer models that ex- plicitly hinge on utterance structures for dia- logue contradiction detection are more robust and generalize well on both analysis and out- of-distribution dialogues than standard (un- structured) Transformers. We also show that our best contradiction detection model corre- lates well with human judgments and further provide evidence for its usage in both auto- matically evaluating and improving the consis- tency of state-of-the-art generative chatbots. 1 Introduction Recent progress on neural approaches to natural language processing (Devlin et al., 2019; Brown et al., 2020), and the availability of large amounts of conversational data (Lowe et al., 2015; Smith et al., 2020) have triggered a resurgent inter- est on building intelligent open-domain chatbots. Newly developed end-to-end neural bots (Zhang et al., 2020; Adiwardana et al., 2020; Roller et al., 2020) are claimed to be superior to their prede- cessors (Worsnick, 2018; Zhou et al., 2020) using various human evaluation techniques (See et al., 2019; Li et al., 2019; Adiwardana et al., 2020) that aim to give a more accurate measure of what makes a good conversation. While the success is indis- putable, there is still a long way to go before we ∗* Dolphins are mammals, not fish. Human Human Human BlenderBot 2.7B Figure 1: Contradictory dialogues contained in our new DECODE dataset. The main train, valid and test sets contain human-written dialogues with deliberate con- tradictions (example at top), and an out-of-domain test set consists of labeled human-bot dialogues (bottom), involving state-of-the-art bots (Roller et al., 2020). arrive at human-like open-domain chatbots. For example, it has been shown that open-domain chat- bots frequently generate annoying errors (Adiwar- dana et al., 2020; Roller et al., 2020) and a notori- ous one among these is the class of contradiction, or consistency errors. When interacting with chatbots, people carry over many of the same expectations as when inter- acting with humans (Nass and Moon, 2000). Self- contradictions by these bots (see Fig.1, bottom) are often jarring, immediately disrupt the conver- 1700 sational flow, and help support arguments about whether generative models could ever really under- stand what they are saying at all (Marcus, 2018). From a listener’s perspective, such inconsistent bots fail to gain …

Content truncated to 500 words