Nonsense!: Quality Control via Two-Step Reason Selection for Annotating Local Acceptability and Related Attributes in News Editorials

Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing, pages 2954–2963, Hong Kong, China, November 3–7, 2019. c⃝2019 Association for Computational Linguistics 2954 Nonsense!: Quality Control via Two-Step Reason Selection for Annotating Local Acceptability and Related Attributes in News Editorials Wonsuk Yang Seungwon Yoon Ada Carpenter Jong C. Park† School of Computing Korea Advanced Institute of Science and Technology {derrick0511, swyoon, ada, park}@nlp.kaist.ac.kr Abstract Annotation quality control is a critical aspect for building reliable corpora through linguistic annotation. In this study, we present a sim- ple but powerful quality control method using two-step reason selection. We gathered sen- tential annotations of local acceptability and three related attributes through a crowdsourc- ing platform. For each attribute, the reason for the choice of the attribute value is selected in a two-step manner. The options given for rea- son selection were designed to facilitate the detection of a nonsensical reason selection. We assume that a reliable annotation may not contain a nonsensical reason selected for the choice of the attribute value, and an annota- tion that contains a nonsensical reason is less reliable than the one without such reason. Our method, based solely on this assumption, is found to retain the annotations with remark- able quality out of the entire annotations mixed with those of low quality. 1 Introduction Crowdsourcing has recently enabled the collection of large amounts of data quickly. However, crowd- sourced annotations require more quality control than traditional in-house annotations. This is be- cause it is difficult to assure the expertise of the participating crowdsource workers in the linguis- tic annotation. To ensure the quality of the annotations ob- tained through crowdsourcing, a frequently uti- lized approach is to request several workers an- notate the same data, aggregate the annotations, and infer the ground truth based on the assumption that the consensus of the workers would lead to the ground truth (Lease, 2011). Various statistical methods have also been proposed to filter out the mistakes or (spamming) random responses of the †Corresponding author crowdsource workers (Liu et al., 2012; Hovy et al., 2013; Nguyen et al., 2017). However, the way to filter out the mistakes or the random responses through statistical means is difficult to utilize for a fundamentally subjective annotation task. This is because statistical methods make it hard to dif- ferentiate the annotator’s mistakes from their sub- jective annotations, as the individual subjectivity of views on different topics can make the gathered data apparently random. Local acceptability, which is for how much the given text is rationally worthy of being believed to be true (Wachsmuth et al., 2017; Yang et al., 2019), is a fundamentally subjective measure. It is thus hard to utilize only statistical methods for the annotation quality control. Yet, local accept- ability is also understood as a strong factor influ- encing the argumentation quality in writing, which is of great importance for computer-assisted writ- ing. In this work, we present a simple …

Content truncated to 500 words