Don’t Say That! Making Inconsistent Dialogue Unlikely with Unlikelihood Training

Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 4715–4728 July 5 - 10, 2020. c⃝2020 Association for Computational Linguistics 4715 Don’t Say That! Making Inconsistent Dialogue Unlikely with Unlikelihood Training Margaret Li1, Stephen Roller1, Ilia Kulikov2⋆, Sean Welleck2⋆ Y-Lan Boureau1, Kyunghyun Cho1,2, Jason Weston1,2 1Facebook AI Research 2New York University {margaretli,roller,ylan,kyunghyuncho,jase}@fb.com wellecks@nyu.edu,kulikov@cs.nyu.edu Abstract Generative dialogue models currently suffer from a number of problems which standard maximum likelihood training does not ad- dress. They tend to produce generations that (i) rely too much on copying from the context, (ii) contain repetitions within utterances, (iii) overuse frequent words, and (iv) at a deeper level, contain logical flaws. In this work we show how all of these problems can be ad- dressed by extending the recently introduced unlikelihood loss (Welleck et al., 2019a) to these cases. We show that appropriate loss functions which regularize generated outputs to match human distributions are effective for the first three issues. For the last important general issue, we show applying unlikelihood to collected data of what a model should not do is effective for improving logical consistency, potentially paving the way to generative mod- els with greater reasoning ability. We demon- strate the efficacy of our approach across sev- eral dialogue tasks. 1 Introduction Open-ended tasks such as dialogue reveal a num- ber of issues with current neural text generation methods. In more strongly grounded tasks such as machine translation and image captioning, current encoder-decoder architectures provide strong per- formance, where mostly word-level decisions are often taken correctly by the model. However, crit- ical failings are exposed in less constrained gener- ation: reliance on repetitive copying and overuse of frequent words, and an inability to maintain logical coherence. The former shows the learn- ing objective is faulty in that it cannot match sim- ple statistics of the training data, while the latter touches more to the heart of artificial intelligence: ⋆Work done while at Facebook AI Research (FAIR). Figure 1: GPT-2 345M model completions can show lack of coherence, e.g. direct contradictions. these models do not understand what they are say- ing. For example, Figure 1 shows how the 345M- parameter GPT2 model (Radford et al., 2019) can give high probability to contradictory generations. In this work, we show how the recently in- troduced unlikelihood objective (Welleck et al., 2019a) can be generalized to remedy these prob- lems. Unlikelihood is a technique developed for removal of repetition in language model comple- tions, and works by adding an extra term to the objective that forces repetitions to have low proba- bility, alleviating the degenerative problems high- lighted in Holtzman et al. (2019). In fact, unlike- lihood can be seen as a much more general frame- work, as we will see. We first generalize unlikelihood to a different domain: dialogue, where we measure statistics of the training distribution in terms of contextual copies, within-utterance repeats, and vocabulary usage. We then develop loss functions that con- trol these statistics, providing improved metrics on several tasks. …

Content truncated to 500 words