An Evaluation Dataset for Intent Classification and Out-of-Scope Prediction

Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing, pages 1311–1316, Hong Kong, China, November 3–7, 2019. c⃝2019 Association for Computational Linguistics 1311 An Evaluation Dataset for Intent Classification and Out-of-Scope Prediction Stefan Larson Anish Mahendran Joseph J. Peper Christopher Clarke Andrew Lee Parker Hill Jonathan K. Kummerfeld Kevin Leach Michael A. Laurenzano Lingjia Tang Jason Mars Clinc, Inc. Ann Arbor, MI, USA stefan@clinc.com Abstract Task-oriented dialog systems need to know when a query falls outside their range of sup- ported intents, but current text classification corpora only define label sets that cover ev- ery example. We introduce a new dataset that includes queries that are out-of-scope— i.e., queries that do not fall into any of the system’s supported intents. This poses a new challenge because models cannot assume that every query at inference time belongs to a system-supported intent class. Our dataset also covers 150 intent classes over 10 domains, capturing the breadth that a production task- oriented agent must handle. We evaluate a range of benchmark classifiers on our dataset along with several different out-of-scope iden- tification schemes. We find that while the clas- sifiers perform well on in-scope intent classi- fication, they struggle to identify out-of-scope queries. Our dataset and evaluation fill an im- portant gap in the field, offering a way of more rigorously and realistically benchmarking text classification in task-driven dialog systems. 1 Introduction Task-oriented dialog systems have become ubiq- uitous, providing a means for billions of people to interact with computers using natural language. Moreover, the recent influx of platforms and tools such as Google’s DialogFlow or Amazon’s Lex for building and deploying such systems makes them even more accessible to various industries and de- mographics across the globe. Tools for developing such systems start by guid- ing developers to collect training data for intent classification: the task of identifying which of a fixed set of actions the user wishes to take based on their query. Relatively few public datasets exist for evaluating performance on this task, and those that do exist typically cover only a very small number of intents (e.g. Coucke et al. (2018), which has 7 What is my balance? You have $1,847.51 across your 3 accounts. How are my sports teams doing? Who has the best record in the NBA? 1 2 3 ✓ ✗ Sorry, I can only answer questions about banking. ✓ Your last payday was on the 1st of November. Figure 1: Example exchanges between a user (blue, right side) and a task-driven dialog system for personal finance (grey, left side). The system correctly identi- fies the user’s query in 1 , but in 2 the user’s query is mis-identified as in-scope, and the system gives an unrelated response. In 3 the user’s query is correctly identified as out-of-scope and the system gives a fall- back response. intents). Furthermore, such resources do not facil- itate analysis of out-of-scope queries: queries that users may reasonably …

Content truncated to 500 words