Learning with Different Amounts of Annotation: From Zero to Many Labels

Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 7620–7632 November 7–11, 2021. c⃝2021 Association for Computational Linguistics 7620 Learning with Different Amounts of Annotation: From Zero to Many Labels Shujian Zhang Chengyue Gong The University of Texas at AustinEunsol Choi szhang19@utexas.edu, {cygong,eunsol}@cs.utexas.edu Abstract Training NLP systems typically assumes ac- cess to annotated data that has a single human label per example. Given imperfect labeling from annotators and inherent ambiguity of lan- guage, we hypothesize that single label is not sufficient to learn the spectrum of language in- terpretation. We explore new annotation distri- bution schemes, assigning multiple labels per example for a small subset of training exam- ples. Introducing such multi label examples at the cost of annotating fewer examples brings clear gains on natural language inference task and entity typing task, even when we simply first train with a single label data and then fine tune with multi label examples. Extend- ing a MixUp data augmentation framework, we propose a learning algorithm that can learn from training examples with different amount of annotation (with zero, one, or multiple la- bels). This algorithm efficiently combines sig- nals from uneven training data and brings ad- ditional gains in low annotation budget and cross domain settings. Together, our method achieves consistent gains in two tasks, suggest- ing distributing labels unevenly among train- ing examples can be beneficial for many NLP tasks.1 1 Introduction Crowdsourcing annotations (Rajpurkar et al., 2016; Bowman et al., 2015) has become a common prac- tice for developing natural language processing benchmark datasets. Even after thorough qual- ity control, it is often infeasible to reach com- plete annotator agreement, as annotators make mis- takes (Freitag et al., 2021) and ambiguity is a key feature of human communication (Asher and Las- carides, 2005). Rich prior works (Passonneau et al., 2012; Pavlick and Kwiatkowski, 2019; Nie et al., 2020; Min et al., 2020; Ferracane et al., 2021) show 1Code and data split is available at https://github. com/szhang42/Uneven_training_data. I have 100 unlabeled examples and a budget to collect 100 labels to build a training corpus. How should I distribute the labels? Default Labeling ................... Uneven Labeling Setting y1 y2 x1 x2 x1 x20 x21 x30 x31 x100 y11 y12 y201 y202 <latexit sha1_base64="PKP/G/k7PwOhGBALkdr6Fd5PpYs=">AB63icbVBNS8NAEJ3Ur1q/qh69LBbBU0m0qMeiF48V7Ae0oWy2m2bp7ibsboQS+he8eFDEq3/Im/GTZuDtj4YeLw3w8y8IOFMG9f9dkpr6xubW+Xtys7u3v5B9fCo+NUEdomMY9VL8CaciZp2zDaS 9RFIuA024wucv97hNVmsXy0UwT6gs8lixkBJtcGiQRG1Zrbt2dA60SryA1KNAaVr8Go5ikgkpDONa67mJ8TOsDCOcziqDVNMEkwke076lEguq/Wx+6wydWEwljZkgbN1d8TGRZaT0VgOwU2kV72cvE/r5+a8MbPmExSQyVZLApTjkyM8sfRiClKDJ9agoli9lZEIqwMTaeig3BW35lXQu6t5V/fKhUWveFnGU4QRO4Rw8uIYm3EML2kAgmd4hTdHOC/Ou/OxaC05xcwx/IHz+QMWJI5H</latexit>φ VL8CaciZp2zDaS9RFIuA024wucv97hNVmsXy0UwT6gs8lixkBJtcGiQRG1Zrbt2dA60SryA1KNAaVr8Go5ikgkpDONa67mJ8TOsDCOcziqDVNMEkwke076lEguq/Wx+6wydWEwljZkgbN1d8TGRZaT0VgOwU2kV72cvE/r5+a8MbPmExSQyVZLApTjkyM8sfRiClKDJ9agoli9lZEIqwMTaeig3BW35lXQu6t5V/fKhUWveFnGU4QRO4Rw8uIYm3EML2kAgmd4hTdHOC/Ou/OxaC05xcwx/IHz+QMWJI5H</latexit>φ<latexit sha1_base64="PKP/G/k7PwOhGBALkdr6Fd5PpYs=">AB63icbVBNS8NAEJ3Ur1q/qh69LBbBU0m0qMeiF48V7Ae0oWy2m2bp7ibsboQS+he8eFDEq3/Im/GTZuDtj4YeLw3w8y8IOFMG9f9dkpr6xubW+Xtys7u3v5B9fCo+NUEdomMY9 y99 y100 x99 x100 ............... ............... ............... Figure 1: Re-thinking how to distribute annotation bud- get. Each blue tag represents a human annotation for the corresponding x. Examples in the orange shaded area are assigned many labels (multi label data), ex- amples in the yellow shaded area are assigned a single label (single label data), and examples in grey shaded area are not assigned any labels. Models trained on a combination of multi label, single label and unlabeled data outperform models trained on single label data on both label accuracy and label distribution metrics for entailment and entity typing task. that disagreement among annotators is not an anno- tation artifact but rather core linguistic phenomena. Despite observing such inherent ambiguity, most …

Content truncated to 500 words