MuTual: A Dataset for Multi-Turn Dialogue Reasoning

Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 1406–1416 July 5 - 10, 2020. c⃝2020 Association for Computational Linguistics 1406 MuTual: A Dataset for Multi-Turn Dialogue Reasoning Leyang Cui†‡∗, Yu Wu3, Shujie Liu3, Yue Zhang‡, Ming Zhou3 †Zhejiang University 3Microsoft Research Asia ‡School of Engineering, Westlake University ‡{cuileyang,zhangyue}@westlake.edu.cn 3{Wu.Yu,shujliu,mingzhou}@microsoft.com Abstract Non-task oriented dialogue systems have achieved great success in recent years due to largely accessible conversation data and the de- velopment of deep learning techniques. Given a context, current systems are able to yield a relevant and fluent response, but sometimes make logical mistakes because of weak rea- soning capabilities. To facilitate the conver- sation reasoning research, we introduce Mu- Tual, a novel dataset for Multi-Turn dialogue Reasoning, consisting of 8,860 manually anno- tated dialogues based on Chinese student En- glish listening comprehension exams. Com- pared to previous benchmarks for non-task ori- ented dialogue systems, MuTual is much more challenging since it requires a model that can handle various reasoning problems. Empiri- cal results show that state-of-the-art methods only reach 71%, which is far behind the hu- man performance of 94%, indicating that there is ample room for improving reasoning abil- ity. MuTual is available at https://github. com/Nealcly/MuTual. 1 Introduction Building an intelligent conversational agent is one of the longest running goals in AI. Existing con- versational agents can be categorized into task- oriented dialogue systems (Kannan et al., 2016) and non-task-oriented chatbot systems (Shum et al., 2018; Wu et al., 2019). Owing to the rise of deep learning techniques and the large amount of conver- sation data for training (Lowe et al., 2015; Wu et al., 2017; Zhang et al., 2018b), we are now witnessing promising results of chatbots both in academia and industry (Pan et al., 2019; Tao et al., 2019). Neural dialogue systems are trained over a large dialogue corpus and used to predict responses given a context. There are two lines of methods. Retrieve- based methods and generation based methods rely ∗Contribution during internship at MSRA. M: Ma'am, you forgot your phone. F: Oh, thanks, I couldn't live without this little thing. M: I know what you mean. It is of great significance to you. So did you enjoy your dinner? F: Oh yes, everything was just perfect. It's so hard to take the whole family out to eat, but your restaurant was perfect. Johnny had his own place to play in and I had time to talk with my sisters and their husbands. ✓ (A) M: Thanks for your compliment for the restaurant. ✘ (B) M: I’m sorry that you don’t have a good time. ✘ (C) M: Goodbye brother! Love you. ✘ (D) M: Hurry up honey, or we will be late for the dinner. Figure 1: B is incorrect because there is no reason to apologize. C and D can be excluded because the rela- tionship between two speakers are waiter and customer based on the context. on matching scores and perplexity scores, respec- tively. Due to the …

Content truncated to 500 words