NIPS Conversational Intelligence Challenge 2017 Winner System: Skill-based Conversational Agent with Supervised Dialog Manager

Proceedings of the 27th International Conference on Computational Linguistics, pages 3681–3692 Santa Fe, New Mexico, USA, August 20-26, 2018. 3681 NIPS Conversational Intelligence Challenge 2017 Winner System: Skill-based Conversational Agent with Supervised Dialog Manager Idris Yusupov 1 and Yurii Kuratov 1, 2 1 Department of Computational Linguistics, Moscow Institute of Physics and Technology 2 Neural Systems and Deep Learning Lab, Moscow Institute of Physics and Technology {i.yusupov, yurii.kuratov}@phystech.edu Abstract We present bot#1337: a dialog system developed for the 1st NIPS Conversational Intelligence Challenge 2017 (ConvAI). The aim of the competition was to implement a bot capable of con- versing with humans based on a given passage of text. To enable conversation, we implemented a set of skills for our bot, including chit-chat, topic detection, text summarization, question answer- ing and question generation. The system has been trained in a supervised setting using a dialogue manager to select an appropriate skill for generating a response. The latter allows a developer to focus on the skill implementation rather than the finite state machine based dialog manager. The proposed system bot#1337 won the competition with an average dialogue quality score of 2.78 out of 5 given by human evaluators. Source code and trained models for the bot#1337 are available on GitHub. 1 Introduction A conversational or a dialogue agent is a system that interacts with a human via voice or text messages. Dialog systems can be task-oriented, i.e. supposed to solve specific tasks such as reserving flight tickets, or general purpose. Another way to differentiate dialog systems is to compare them by domain: closed or open one. When open-domain dialog systems cover a wide range of supported topics, closed-domain are usually specialized on a few topics. The early dialog systems used a rule-based approach to control dialog flow (Weizenbaum, 1966; Wal- lace, 2009). Rule-based systems are hard to maintain and new rules should be created for each new domain. This approach is still widely used in cases where the full control of dialog is crucial (e.g., pro- duction systems). The availability of dialog datasets (DSTC (Williams et al., 2016), Twitter Corpus (Rit- ter et al., 2010), Ubuntu Dialogue Corpus (Lowe et al., 2015)) makes it possible to train end-to-end dialog systems (Sordoni et al., 2015; Vinyals and Le, 2015). End-to-end systems are usually based on recurrent neural networks (Graves, 2013) and sequence-to-sequence models (Sutskever et al., 2014), and use raw dialogues as a training data. One of the main challenges in the field of open-domain dialogue systems is their evaluation. The reason for that is the absence of good evaluation metrics. While goal-oriented systems can be evaluated with the percentage of dialogues when user’s task was accomplished, there is no formal measure of the dialogue quality for open-domain systems. The goal of such systems is to generate responses which suits the context. The quality of the response can be measured by its perplexity measure (Serban et al., 2015) which, however, can not assess the adequacy of an answer. In order to measure it …

Content truncated to 500 words