Dialogue Natural Language Inference

Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 3731–3741 Florence, Italy, July 28 - August 2, 2019. c⃝2019 Association for Computational Linguistics 3731 Dialogue Natural Language Inference Sean Welleck New York University wellecks@nyu.edu Jason Weston Facebook AI Research New York University Arthur Szlam Facebook AI Research Kyunghyun Cho New York University Facebook AI Research CIFAR Azrieli Global Scholar Abstract Consistency is a long standing issue faced by dialogue models. In this paper, we frame the consistency of dialogue agents as natural lan- guage inference (NLI) and create a new natu- ral language inference dataset called Dialogue NLI. We propose a method which demon- strates that a model trained on Dialogue NLI can be used to improve the consistency of a dialogue model, and evaluate the method with human evaluation and with automatic metrics on a suite of evaluation sets designed to mea- sure a dialogue model’s consistency. 1 Introduction A long standing issue faced by dialogue models is consistency (Li et al., 2016; Vinyals et al., 2015; Zhang et al., 2018). An example from (Vinyals et al., 2015) shows a two-round dialogue in which their neural sequence model first responds to what is your job? with i’m a lawyer, then responds to what do you do? with i’m a doctor. Even when in- consistencies are relatively rare and semantically plausible, they are jarring, and because semantic plausibility is not enough to root them out, pre- venting them is challenging. One approach to increasing the consistency of a chit-chat dialogue model was proposed in (Zhang et al., 2018), where the dialogue agent was given a set of personal facts describing its character (a persona) and produces utterances that reflect the persona. The intended outcome is that the agent produces utterances consistent with its given per- sona. However, these models still face the consis- tency issue, as shown in Figure 1. Separately, the framework of Natural Language Inference (NLI) (Bowman et al., 2015; Dagan et al., 2006; Maccartney and Manning, 2009) in- volves learning a mapping between a sentence pair and an entailment category. It is hypothesized that the NLI task is a proxy for general goals in natu- ral language processing, such as language under- standing (Bowman et al., 2015; Williams et al., 2018). Thus, the NLI task has been used for learning general sentence representations (Con- neau et al., 2017) and for evaluating NLP mod- els (Poliak et al., 2018a; Wang et al., 2018), with the expectation that such models will be useful in downstream tasks. Despite this expectation, leveraging an NLI model for a downstream task remains an under- explored research direction. An NLI model may improve downstream task performance if prop- erly used, while downstream tasks may yield new datasets or identify issues with existing NLI mod- els, thus expanding the NLI research domain. In this paper, we reduce the problem of consis- tency in dialogue to natural language inference. We first create a dataset, Dialogue NLI,1 which contains sentence pairs labeled as …

Content truncated to 500 words