Bootstrapping incremental dialogue systems from minimal data: the generalisation power of dialogue grammars

Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 2220–2230 Copenhagen, Denmark, September 7–11, 2017. c⃝2017 Association for Computational Linguistics Bootstrapping incremental dialogue systems from minimal data: the generalisation power of dialogue grammars Arash Eshghi Interaction Lab Heriot-Watt University a.eshghi@hw.ac.uk Igor Shalyminov Interaction Lab Heriot-Watt University is33@hw.ac.uk Oliver Lemon Interaction Lab Heriot-Watt University o.lemon@hw.ac.uk Abstract We investigate an end-to-end method for automatically inducing task-based dia- logue systems from small amounts of unannotated dialogue data. It combines an incremental semantic grammar - Dynamic Syntax and Type Theory with Records (DS-TTR) - with Reinforcement Learn- ing (RL), where language generation and dialogue management are a joint deci- sion problem. The systems thus pro- duced are incremental: dialogues are pro- cessed word-by-word, shown previously to be essential in supporting natural, spon- taneous dialogue. We hypothesised that the rich linguistic knowledge within the grammar should enable a combinatorially large number of dialogue variations to be processed, even when trained on very few dialogues. Our experiments show that our model can process 74% of the Facebook AI bAbI dataset even when trained on only 0.13% of the data (5 dialogues). It can in addition process 65% of bAbI+, a cor- pus1 we created by systematically adding incremental dialogue phenomena such as restarts and self-corrections to bAbI. We compare our model with a state-of-the- art retrieval model, memn2n(Bordes et al., 2017). We find that, in terms of semantic accuracy, memn2nshows very poor robust- ness to the bAbI+ transformations even when trained on the full bAbI dataset. 1 Introduction There are currently several key problems for the practical data-driven (rather than hand-crafted) development of task-oriented dialogue systems, 1Dataset available at https://bit.ly/babi_plus among them: (1) large amounts of dialogue data are needed, i.e. thousands of examples in a do- main; (2) this data is usually required to be anno- tated with task-specific semantic/pragmatic infor- mation for the domain (e.g. various dialogue act schemes); and (3) the resulting systems are gen- erally turn-based, and so do not support natural spontaneous dialogue which is processed incre- mentally, word-by-word, with many characteristic phenomena that arise from this incrementality. In overcoming issue (2), a recent advance made in research on (non-task) chat dialogues has been the development of so-called “end-to-end" sys- tems, in which all components are trained from textual dialogue examples, e.g. (Sordoni et al., 2015; Vinyals and Le, 2015). However, as Bordes and Weston (2017) argued, these end-to-end meth- ods may not transfer well to task-based settings (where the user is trying to achieve a domain goal, such as booking a flight or finding a restaurant, re- sulting in an API call). Bordes and Weston (2017) then presented an end-to-end method using Mem- ory Networks (memn2ns), which achieves 100% performance on a test-set of 1000 dialogues, af- ter being trained on 1000 dialogues. This method processes dialogues turn-by-turn, and so does not have the advantages of more natural incremen- tal systems (Aist et al., 2007; Skantze and Hjal- marsson, 2010); nor does it really perform lan- …

Content truncated to 500 words