Rethinking Annotation: Can Language Learners Contribute?

Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics Volume 1: Long Papers, pages 14714–14733 July 9-14, 2023 ©2023 Association for Computational Linguistics Rethinking Annotation: Can Language Learners Contribute? Haneul Yoo1, Rifki Afina Putri1, Changyoon Lee1, Youngin Lee1, So-Yeon Ahn1, Dongyeop Kang2, Alice Oh1 1KAIST, South Korea, 2University of Minnesota, USA {haneul.yoo, rifkiaputri, cyoon47, conviette}@kaist.ac.kr, ahnsoyeon@kaist.ac.kr, dongyeop@umn.edu, alice.oh@kaist.edu Abstract Researchers have traditionally recruited native speakers to provide annotations for widely used benchmark datasets. However, there are lan- guages for which recruiting native speakers can be difficult, and it would help to find learners of those languages to annotate the data. In this paper, we investigate whether language learners can contribute annotations to bench- mark datasets. In a carefully controlled an- notation experiment, we recruit 36 language learners, provide two types of additional re- sources (dictionaries and machine-translated sentences), and perform mini-tests to measure their language proficiency. We target three lan- guages, English, Korean, and Indonesian, and the four NLP tasks of sentiment analysis, nat- ural language inference, named entity recog- nition, and machine reading comprehension. We find that language learners, especially those with intermediate or advanced levels of lan- guage proficiency, are able to provide fairly accurate labels with the help of additional re- sources. Moreover, we show that data annota- tion improves learners’ language proficiency in terms of vocabulary and grammar. One impli- cation of our findings is that broadening the an- notation task to include language learners can open up the opportunity to build benchmark datasets for languages for which it is difficult to recruit native speakers. 1 Introduction Data annotation is important, and in NLP, it has been customary to recruit native speakers of the target languages, even though it is difficult to re- cruit native speakers for many languages. Mean- while, there are many people learning another lan- guage, for instance, Duolingo claims that 1.8 bil- lion people are learning a foreign language using their app. 1 In this paper, we examine whether language learners can annotate data as well as native speak- 1https://www.duolingo.com/ Figure 1: Recruiting language learners in NLP data annotation. They can be assisted by dictionaries or MT systems. ers and whether their annotations can be used to train language models. We explore this question with five control variables that may affect the qual- ity of language learner annotations. These are the language, task, learners’ language proficiency, difficulty of the annotation questions, and addi- tional resources that learners can consult. We re- cruited learners at various levels of proficiency in English (high-resource), Korean (mid-resource), and Indonesian (low-resource). They annotated data on four tasks, sentiment analysis (SA), natural language inference (NLI), named entity recogni- tion (NER), and machine reading comprehension (MRC). We ask three levels of learners to complete multiple sessions of data annotation given with the help of a dictionary or machine-translated texts. Our major findings, both in terms of the qual- ity and learning effect of learners’ annotations, are summarized as follows: We measure the de- gree of inter-annotator …

Content truncated to 500 words