Data Query Language and Corpus Tools for Slot-Filling and Intent Classification Data

Proceedings of the 12th Conference on Language Resources and Evaluation (LREC 2020), pages 7060–7068 Marseille, 11–16 May 2020 c⃝ European Language Resources Association (ELRA), licensed under CC-BY-NC 7060 Data Query Language and Corpus Tools for Slot-Filling and Intent Classification Data Stefan Larson, Eric Guldan, Kevin Leach Clinc, Inc. Ann Arbor, Michigan, USA stefan@clinc.com Abstract Typical machine learning approaches to developing task-oriented dialog systems require the collection and management of large amounts of training data, especially for the tasks of intent classification and slot-filling. Managing this data can be cumbersome without dedicated tools to help the dialog system designer understand the nature of the data. This paper presents a toolkit for analyzing slot-filling and intent classification corpora. We present a toolkit that includes (1) a new lightweight and readable data and file format for intent classification and slot-filling corpora, (2) a new query language for searching intent classification and slot-filling corpora, and (3) tools for understanding the structure and makeup for such corpora. We apply our toolkit to several well-known NLU datasets, and demonstrate that our toolkit can be used to uncover interesting and surprising insights. By releasing our toolkit to the research community, we hope to enable others to develop more robust and intelligent slot-filling and intent classification models. Keywords: corpus, conversational systems/dialogue/chatbots/human-robot interaction, tools 1. Introduction Contemporary task-oriented dialog systems rely on natural language understanding (NLU) models to classify a user’s utterance into an intent class (e.g., “what is my savings account balance” → get balance) as well as to extract important entities from the utterance (e.g., “savings ac- count” → BANK ACCOUNT). While the choice of particular NLU models for the tasks of intent classification and slot- filling (also called slot or entity extraction) can depend on factors such as the task domain, deployment environment, and computational constraints, one thing that all machine learning-driven NLU models have in common is that they require large amounts of carefully annotated and curated training data. Moreover, developing, debugging, improv- ing, and maintaining such models in production environ- ments requires rapidly searching through training corpora to gain insights from the data. Such insights can help the dialog system designer spot gaps and errors in the training data, potentially improving system performance drastically. As a motivating example, suppose a system designer/main- tainer seeks to train an NLU system composed of intent classification and slot-filling models. Given a dataset anno- tated in a style similar to that shown in Figure 1, there are many valuable insights that the designer/maintainer might wish to understand, including: A. The distribution of slot values for a given slot type. If a particular slot type has only a few unique slot values in a training set, then a slot-filling model trained on this data will likely be overfit to those few values. B. The composition and distribution of context words for a particular slot type. For example, in the training ut- terance book me a flight from detroit ORIGIN to new york TARGET the left and right context 1-grams …

Content truncated to 500 words