Bridging the Gap between Language Models and Cross-Lingual Sequence Labeling

Two Contrasting Data Annotation Paradigms for Subjective NLP Tasks Paul Röttger1,2, Bertie Vidgen2, Dirk Hovy3, and Janet B. Pierrehumbert1 1University of Oxford 2The Alan Turing Institute 3Bocconi University Abstract Labelled data is the foundation of most natural language processing tasks. However, labelling data is difficult and there often are diverse valid beliefs about what the correct data labels should be. So far, dataset creators have ac- knowledged annotator subjectivity, but rarely actively managed it in the annotation process. This has led to partly-subjective datasets that fail to serve a clear downstream use. To ad- dress this issue, we propose two contrasting paradigms for data annotation. The descrip- tive paradigm encourages annotator subjectiv- ity, whereas the prescriptive paradigm discour- ages it. Descriptive annotation allows for the surveying and modelling of different beliefs, whereas prescriptive annotation enables the training of models that consistently apply one belief. We discuss benefits and challenges in implementing both paradigms, and argue that dataset creators should explicitly aim for one or the other to facilitate the intended use of their dataset. Lastly, we conduct an annotation experiment using hate speech data that illus- trates the contrast between the two paradigms. 1 Introduction Many natural language processing (NLP) tasks are subjective, in the sense that there are diverse valid beliefs about what the correct data labels should be. Some tasks, like hate speech detection, are highly subjective: different people have very dif- ferent beliefs about what should or should not be labelled as hateful (Talat, 2016; Salminen et al., 2019; Davani et al., 2021a), and while some be- liefs are more widely accepted than others, there is no single objective truth. Other examples in- clude toxicity (Sap et al., 2019, 2021), harassment (Al Kuwatly et al., 2020), harmful content (Jiang et al., 2021) and stance detection (Luo et al., 2020; AlDayel and Magdy, 2021) as well as sentiment analysis (Kenyon-Dean et al., 2018; Poria et al., 2020). But even for seemingly objective tasks like Figure 1: Two key questions for dataset creators. part-of-speech tagging, there is subjective disagree- ment between annotators (Plank et al., 2014b). In this article, we argue that dataset creators should consider the role of annotator subjectivity in the annotation process and either explicitly en- courage it or discourage it. Annotators may subjec- tively disagree about labels (e.g., for hate speech) but dataset creators can and should decide, based on the intended downstream use of their dataset, whether they want to a) capture different beliefs or b) encode one specific belief in their data. As a framework, we propose two contrasting data annotation paradigms. Each paradigm facil- itates a clear and distinct downstream use. The descriptive paradigm encourages annotator sub- jectivity to create datasets as granular surveys of individual beliefs. Descriptive data annotation thus allows for the capturing and modelling of different beliefs. The prescriptive paradigm, on the other hand, discourages annotator subjectivity and in- stead tasks annotators with encoding one specific belief, formulated in the annotation guidelines. Pre- scriptive data annotation thus enables the …

Content truncated to 500 words