Inconsistencies in Crowdsourced Slot-Filling Annotations: A Typology and Identification Methods

Proceedings of the 28th International Conference on Computational Linguistics, pages 5035–5046 Barcelona, Spain (Online), December 8-13, 2020 5035 Inconsistencies in Crowdsourced Slot-Filling Annotations: A Typology and Identification Methods Stefan Larson ♣∗† Adrian Cheung♦ Anish Mahendran ♠† Kevin Leach ♠† Jonathan K. Kummerfeld ♠† ♣Rosegold AI ♦Clinc, Inc. ♠University of Michigan Ann Arbor, MI, USA Ann Arbor, MI, USA Ann Arbor, MI, USA Abstract Slot-filling models in task-driven dialog systems rely on carefully annotated training data. How- ever, annotations by crowd workers are often inconsistent or contain errors. Simple solutions like manually checking annotations or having multiple workers label each sample are expensive and waste effort on samples that are correct. If we can identify inconsistencies, we can focus effort where it is needed. Toward this end, we define six inconsistency types in slot-filling annotations. Using three new noisy crowd-annotated datasets, we show that a wide range of inconsistencies occur and can impact system performance if not addressed. We then introduce automatic meth- ods of identifying inconsistencies. Experiments on our new datasets show that these methods effectively reveal inconsistencies in data, though there is further scope for improvement. 1 Introduction Slot-filling is a key component of task-driven dialog systems, providing a way for systems to extract key properties from user queries. For example, a slot-filling model can extract the tokens “New York” as a TO LOCATION slot in the query “book a flight to New York”. Standard slot-filling models train or finetune on large datasets of carefully-annotated data that is domain specific. This means there is an ongoing need for annotating new datasets for new domains. Typically, data annotation is completed via small tasks with brief instructions completed by non-expert crowd workers. Collecting high quality data in this setting is challenging, involving multiple rounds of pilot annotations and analysis to develop suitable instructions (Alonso et al., 2015). Figure 1 shows examples of inconsistencies in annotations performed by crowd workers. Training on data with these issues will lead to lower quality models, which in turn decrease the effectiveness of the overall dialog system. Most research on improving data quality has focused on mechanisms such as aggregation (Parde and Nielsen, 2017), worker filtering (Li and Liu, 2015), and attention checks (Oppenheimer et al., 2009). These all raise costs and primarily address clear inconsistencies (such as in examples 1, 4, 5, and 6) but not more subtle cases like the inclusion of “dollar” in examples 2 and 3. Additionally, annotation-as- a-service (e.g., scale.ai) is being widely used by developers but often cannot be customized by the developer to add these kinds of mechanisms. If we could identify all of these inconsistencies then we could fix them without expending effort on examples that were correctly annotated and we could improve task instructions to reduce further inconsistencies. In this paper, we catalog a typology of annotation inconsistencies that occur in slot-filling data and present several automatic methods for identifying inconsistencies. To demonstrate our ideas, we intro- duce and analyze three new crowd-annotated datasets.1 By analyzing the …

Content truncated to 500 words