Iterative Feature Mining for Constraint-Based Data Collection to Increase Data Diversity and Model Robustness

Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing, pages 8097–8106, November 16–20, 2020. c⃝2020 Association for Computational Linguistics 8097 Iterative Feature Mining for Constraint-Based Data Collection to Increase Data Diversity and Model Robustness Stefan Larson∗♣† Anthony Zheng♠† Anish Mahendran♠† Rishi Tekriwal♠† Adrian Cheung♦ Eric Guldan♠† Kevin Leach♠† Jonathan K. Kummerfeld♠† ♣ Rosegold AI ♠ University of Michigan ♦Clinc, Inc. Ann Arbor, MI, USA Ann Arbor, MI, USA Ann Arbor, MI, USA Abstract Diverse data is crucial for training robust mod- els, but crowdsourced text often lacks diversity as workers tend to write simple variations from prompts. We propose a general approach for guiding workers to write more diverse text by iteratively constraining their writing. We show how prior workflows are special cases of our approach, and present a way to apply the ap- proach to dialog tasks such as intent classifica- tion and slot-filling. Using our method, we cre- ate more challenging versions of test sets from prior dialog datasets and find dramatic perfor- mance drops for standard models. Finally, we show that our approach is complementary to recent work on improving data diversity, and training on data collected with our approach leads to more robust models. 1 Introduction Crowdsourcing is widely used to collect data, in- cluding cases where workers are writing new text, such as questions (Rajpurkar et al., 2016), dialog (Budzianowski et al., 2018), and captions (Rus- sakovsky et al., 2015). To avoid repetition of short labels for images, von Ahn and Dabbish (2004) proposed using a taboo list, preventing workers from writing labels that previous workers had writ- ten. This idea has since been applied to emotion annotation (Pearl and Steyvers, 2010) and word association (Vickrey et al., 2008; Lafourcade and Joubert, 2012). However, in all of these cases the constraint is that there cannot be an exact match with another label. This limits the approach to tasks where workers write a single word or a short phrase. Meanwhile, recent work on dialog has found that crowdsourced data can have limited diversity (Jiang et al., 2017; Kang et al., 2018; Larson et al., 2019a). This limited diversity has dramatic consequences, as models trained on such data may not generalize well to unseen or uncommon inputs. ∗Corresponding email: stefan.dataset@gmail.com. †Work performed while author was employed by Clinc. We present a generalization of the taboo list idea that can be applied to longer text like sentences. First, rather than features in the taboo list being complete labels, we allow them to be anything, e.g., for intent classification, each feature in the list is a single word that the worker cannot use in their new utterance. To create the taboo list, we propose using a simple model to find over-represented features in the data collected so far. Second, rather than having a 1-1 mapping of taboo lists to examples we allow any mapping, e.g., for intent classification we have a taboo list for each intent. To show how this idea improves diversity for longer text, …

Content truncated to 500 words