Do Neural Dialog Systems Use the Conversation History Effectively? An Empirical Study

Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 32–37 Florence, Italy, July 28 - August 2, 2019. c⃝2019 Association for Computational Linguistics 32 Do Neural Dialog Systems Use the Conversation History Effectively? An Empirical Study Chinnadhurai Sankar1,2,4∗ Sandeep Subramanian1,2,5 Christopher Pal1,3,5 Sarath Chandar1,2,4 Yoshua Bengio 1,2 1Mila 2Universit´e de Montr´eal 4Google Research, Brain Team3 ´Ecole Polytechnique de Montr´eal 5Element AI, Montr´eal Abstract Neural generative models have been become increasingly popular when building conversa- tional agents. They offer flexibility, can be eas- ily adapted to new domains, and require min- imal domain engineering. A common criti- cism of these systems is that they seldom un- derstand or use the available dialog history ef- fectively. In this paper, we take an empiri- cal approach to understanding how these mod- els use the available dialog history by study- ing the sensitivity of the models to artificially introduced unnatural changes or perturbations to their context at test time. We experiment with 10 different types of perturbations on 4 multi-turn dialog datasets and find that com- monly used neural dialog architectures like re- current and transformer-based seq2seq models are rarely sensitive to most perturbations such as missing or reordering utterances, shuffling words, etc. Also, by open-sourcing our code, we believe that it will serve as a useful diag- nostic tool for evaluating dialog systems in the future 1. 1 Introduction With recent advancements in generative models of text (Wu et al., 2016; Vaswani et al., 2017; Rad- ford et al., 2018), neural approaches to building chit-chat and goal-oriented conversational agents (Sordoni et al., 2015; Vinyals and Le, 2015; Ser- ban et al., 2016; Bordes and Weston, 2016; Serban et al., 2017b) has gained popularity with the hope that advancements in tasks like machine transla- tion (Bahdanau et al., 2015), abstractive summa- rization (See et al., 2017) should translate to dialog systems as well. While these models have demon- strated the ability to generate fluent responses, ∗Corresponding author: chinnadhurai@gmail.com 1Code is available at https://github.com/ chinnadhurai/ParlAI/ they still lack the ability to “understand” and pro- cess the dialog history to produce coherent and interesting responses. They often produce bor- ing and repetitive responses like “Thank you.” (Li et al., 2015; Serban et al., 2017a) or meander away from the topic of conversation. This has been often attributed to the manner and extent to which these models use the dialog history when generating re- sponses. However, there has been little empirical investigation to validate these speculations. In this work, we take a step in that direction and confirm some of these speculations, showing that models do not make use of a lot of the informa- tion available to it, by subjecting the dialog his- tory to a variety of synthetic perturbations. We then empirically observe how recurrent (Sutskever et al., 2014) and transformer-based (Vaswani et al., 2017) sequence-to-sequence (seq2seq) models re- spond to these changes. The central premise of this work is that models make minimal use of cer- tain types of information …

Content truncated to 500 words