Analysing the potential of seq-to-seq models for incremental interpretation in task-oriented dialogue

Proceedings of the 2018 EMNLP Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP, pages 165–174 Brussels, Belgium, November 1, 2018. c⃝2018 Association for Computational Linguistics 165 Analysing the potential of seq-to-seq models for incremental interpretation in task-oriented dialogue Dieuwke Hupkes, Sanne Bouwmeester, and Raquel Fern´andez Institute for Logic Language and Computation University of Amsterdam {d.hupkes,raquel.fernandez}@uva.nl sanne.bouwmeester1@student.uva.nl Abstract We investigate how encoder-decoder models trained on a synthetic dataset of task-oriented dialogues process disfluencies, such as hesita- tions and self-corrections. We find that, con- trary to earlier results, disfluencies have very little impact on the task success of seq-to-seq models with attention. Using visualisations and diagnostic classifiers, we analyse the rep- resentations that are incrementally built by the model, and discover that models develop little to no awareness of the structure of disfluen- cies. However, adding disfluencies to the data appears to help the model create clearer repre- sentations overall, as evidenced by the atten- tion patterns the different models exhibit. 1 Introduction The use of Recurrent Neural Networks (RNNs) to tackle sequential language tasks has become stan- dard in natural language processing, after impres- sive accomplishments in speech recognition, ma- chine translation, and entailment (e.g., Sutskever et al., 2014; Bahdanau et al., 2015b; Kalchbren- ner et al., 2014). Recently, RNNs have also been exploited as tools to model dialogue sys- tems. Inspired by neural machine translation, re- searchers such as Ritter et al. (2011) and Vinyals and Le (2015) pioneered an approach to open- domain chit-chat conversation based on sequence- to-sequence models (Sutskever et al., 2014). In this paper, we focus on task-oriented dialogue, where the conversation serves to fulfil an inde- pendent goal in a given domain. Current neural dialogue models for task-oriented dialogue tend to equip systems with external memory compo- nents (Bordes et al., 2017), since key information needs to be stored for potentially long time spans. One of our goals here is to analyse to what ex- tent sequence-to-sequence models without exter- nal memory can deal with this challenge. In addition, we consider language realisations that include disfluencies common in dialogue in- teraction, such as repetitions and self-corrections (e.g., I’d like to make a reservation for six, I mean, for eight people). Disfluencies have been investigated extensively in psycholinguistics, with a range of studies showing that they affect sen- tence processing in intricate ways (Levelt, 1983; Fox Tree, 1995; Bailey and Ferreira, 2003; Fer- reira and Bailey, 2004; Lau and Ferreira, 2005; Brennan and Schober, 2001). Most computational work on disfluencies, however, has focused on de- tection rather than on disfluency processing and interpretation (e.g., Stolcke and Shriberg, 1996; Heeman and Allen, 1999; Zwarts et al., 2010; Qian and Liu, 2013; ?; ?). In contrast, our aim is to get a better understanding of how RNNs process dis- fluent utterances and to analyse the impact of such disfluencies on a downstream task—in this case, issuing an API request reflecting the preferences of the user in a task-oriented dialogue. For our experiments, we use the synthetic dataset …

Content truncated to 500 words