Outlier Detection for Improved Data Quality and Diversity in Dialog Systems

Proceedings of NAACL-HLT 2019, pages 517–527 Minneapolis, Minnesota, June 2 - June 7, 2019. c⃝2019 Association for Computational Linguistics 517 Outlier Detection for Improved Data Quality and Diversity in Dialog Systems Stefan Larson Anish Mahendran Andrew Lee Jonathan K. Kummerfeld Parker Hill Michael A. Laurenzano Johann Hauswald Lingjia Tang Jason Mars Clinc, Inc. Ann Arbor, MI, USA {stefan,anish,andrew,jkk}@clinc.com {parkerhh,mike,johann,lingjia,jason}@clinc.com Abstract In a corpus of data, outliers are either errors: mistakes in the data that are counterproduc- tive, or are unique: informative samples that improve model robustness. Identifying out- liers can lead to better datasets by (1) remov- ing noise in datasets and (2) guiding collection of additional data to fill gaps. However, the problem of detecting both outlier types has re- ceived relatively little attention in NLP, partic- ularly for dialog systems. We introduce a sim- ple and effective technique for detecting both erroneous and unique samples in a corpus of short texts using neural sentence embeddings combined with distance-based outlier detec- tion. We also present a novel data collection pipeline built atop our detection technique to automatically and iteratively mine unique data samples while discarding erroneous samples. Experiments show that our outlier detection technique is effective at finding errors while our data collection pipeline yields highly di- verse corpora that in turn produce more robust intent classification and slot-filling models. 1 Introduction High-quality annotated data is one of the fun- damental drivers of progress in Natural Lan- guage Processing (e.g. Marcus et al., 1993; Koehn, 2005). In order to be effective at producing an accurate and robust model, a dataset needs to be correct while also diverse enough to cover the full range of ways in which the phenomena it targets occur. Substantial research effort has considered dataset correctness (Eskin, 2000; Dickinson and Meurers, 2003; Rehbein and Ruppenhofer, 2017), particularly for crowdsourcing (Snow et al., 2008; Jiang et al., 2017), but addressing diversity in data has received less attention, with the exception of using data from diverse domains (Hovy et al., 2006). Outlier detection, the task of finding ex- amples in a dataset that are atypical, provides a means of approaching the questions of correctness and diversity, but has mainly been studied at the document level (Guthrie et al., 2008; Zhuang et al., 2017), whereas texts in dialog systems are often no more than a few sentences in length. We propose a novel approach that uses sen- tence embeddings to detect outliers in a corpus of short texts. We rank samples based on their dis- tance from the mean embedding of the corpus and consider samples farthest from the mean outliers. Outliers come in two varieties: (1) errors, sen- tences that have been mislabeled whose inclusion in the dataset would be detrimental to model per- formance, and (2) unique samples, sentences that differ in structure or content from most in the data and whose inclusion would be helpful for model robustness. Building upon this approach, we pro- pose a novel crowdsourcing pipeline that distin- guishes errors from unique samples …

Content truncated to 500 words