Improving Human-Labeled Data through Dynamic Automatic Conflict Resolution

Proceedings of the 28th International Conference on Computational Linguistics, pages 3547–3557 Barcelona, Spain (Online), December 8-13, 2020 3547 Improving Human-Labeled Data through Dynamic Automatic Conflict Resolution David Q. Sun*, Hadas Kotek*, Christopher Klein, Mayank Gupta, William Li, Jason D. Williams Apple One Apple Park Way Cupertino, California 95014 dqs@apple.com Abstract This paper develops and implements a scalable methodology for (a) estimating the noisiness of labels produced by a typical crowdsourcing semantic annotation task, and (b) reducing the result- ing error of the labeling process by as much as 20-30% in comparison to other common labeling strategies. Importantly, this new approach to the labeling process, which we name Dynamic Automatic Conflict Resolution (DACR), does not require a ground truth dataset and is instead based on inter-project annotation inconsistencies. This makes DACR not only more accurate but also available to a broad range of labeling tasks. In what follows we present results from a text classification task performed at scale for a commercial personal assistant, and evaluate the inherent ambiguity uncovered by this annotation strategy as compared to other common labeling strategies. 1 Introduction Ground truth datasets for supervised learning are commonly collected through crowdsourcing annotation projects, either on public platforms (e.g. Yandex.Toloka, Amazon Mechanical Turk, Figure Eight) or on in-house counterparts. In both scenarios, the annotation project is conducted by first providing workers with annotation guidelines and then assigning each one with a set of annotation requests. Each unique annotation request is usually assigned to more than one worker, to increase the reliability of the results. Subsequently, some post-processing technique is used to create a consensus label that is recorded as the ground truth. However, since human annotations are intrinsically noisy and subject to the influence of external factors, such a process is unlikely to produce accurate ground truth labels for 100% of the requests. Moreover, the practice of using a singular consensus label for multi-annotated requests mandates a one and only correct interpretation for every request in any given task, effectively denying the existence of potential ambiguities. This implicit denial of ambiguity, while acceptable for certain tasks such as arith- metics, can be problematic in tasks such as natural language understanding (Dumitrache et al., 2018a). Perhaps because of these difficulties, estimating the latent accuracy of any annotation process has been a relatively under-examined subject. In this paper, we first introduce a scalable methodology to estimate the latent inaccuracy of any crowd- sourcing annotation process. The estimation method does not require any preexisting ground truth dataset, and is therefore immune to the Gold Question Inference type of attacks described by Checco et al. (2018). We construct an agent-based model to examine the impact of different annotation strategies on overall annotation accuracy. Inspired by the findings of our simulation experiments, we propose and implement a novel process of Dynamic Automatic Conflict Resolution (DACR), designed to improve the accuracy of the annotation output with limited incremental workload while collecting sufficient annota- tion input for requests with ambiguity. We present annotation results from a …

Content truncated to 500 words