Learning from Measurements in Crowdsourcing Models: Inferring Ground Truth from Diverse Annotation Types

Proceedings of the 27th International Conference on Computational Linguistics, pages 1694–1704 Santa Fe, New Mexico, USA, August 20-26, 2018. 1694 Learning from Measurements in Crowdsourcing Models: Inferring Ground Truth from Diverse Annotation Types Paul Felt IBM Watson∗ plfelt@us.ibm.com Eric K. Ringger Zillow∗ ericri@zillow.com Jordan Boyd-Graber Computer Science, iSchool, UMIACS, and Language Science, University of Maryland jbg@umiacs.umd.edu Kevin Seppi Dept. of Computer Science Brigham Young University kseppi@byu.edu Abstract Annotated corpora enable supervised machine learning and data analysis. To reduce the cost of manual annotation, tasks are often assigned to internet workers whose judgments are reconciled by crowdsourcing models. We approach the problem of crowdsourcing using a framework for learning from rich prior knowledge, and we identify a family of crowdsourcing models with the novel ability to combine annotations with differing structures: e.g., document labels and word labels. Annotator judgments are given in the form of the predicted expected value of measure- ment functions computed over annotations and the data, unifying annotation models. Our model, a specific instance of this framework, compares favorably with previous work. Furthermore, it enables active sample selection, jointly selecting annotator, data item, and annotation structure to reduce annotation effort. Annotierte Korpora erm¨oglichen ¨uberwachtes maschinelles Lernen und Datenanalyse. Um die Kosten f¨ur manuelle Annotationen zu vermeiden, werden Aufgaben h¨aufig Internetarbeitern zugewiesen, deren Urteile durch Crowdsourcing-Modelle abgeglichen werden. Wir n¨ahern uns dem Problem des Crowdsourcings, indem wir einen Rahmen f¨ur das Lernen aus reichem Vor- wissen vorschlagen, und wir bestimmen eine Familie von Crowdsourcing-Modellen mit der F¨ahigkeit, Annotationen mit unterschiedlichen Strukturen zu kombinieren: z.B., Dokument- bezeichnungen und Wortbezeichnungen. Bewertungen werden in Form des vorhergesagten erwarteten Werts von Messfunktionen (measurement functions) gegeben, die ¨uber Annotatio- nen und die Daten berechnet werden. Darin werden die vorherige Annotationsmodelle verein- heitlicht. Unser Modell, eine spezifische Instanz dieses Rahmens, schneidet im Vergleich zu fr¨uheren Arbeiten positiv ab. Dar¨uber hinaus erm¨oglicht es die aktive Stichprobenauswahl, in- dem Kommentator, Datenelement, und Annotationsstruktur gemeinsam ausgew¨ahlt werden, um den Annotationskosten zu reduzieren. 1 Introduction Supervised machine learning is data hungry: new approaches require massive training sets. These train- ing sets can come from inexpensive crowdsourcing platforms, but consistency is often sacrificed for speed and thrift. Sophisticated models (Surowiecki, 2005; Snow et al., 2008; Jurgens, 2013) can over- come the intrinsic annotation noise by reconciling redundant annotation and predicting true labels by modeling the error patterns associated with individual labels, documents, or annotators. These models have typically assumed that we collect document-level labels and nothing else from annotators. But crowd workers could provide other valuable information. For example, if we wanted to predict the sentiment of documents about the weather (Figure 1), intuition says that words like “sunny”, This work is licensed under a Creative Commons Attribution 4.0 International License. License details: http://creativecommons.org/licenses/by/4.0/ ∗ This work was completed while the first and second authors were at Brigham Young University. 1695 • . . . Sunshine: 60s and partly sunny? OK! • . . . shopping, sunshine, margaritas, happiness! • Going to the zoo . . . the weather is …

Content truncated to 500 words