A Needle in a Haystack: An Analysis of High-Agreement Workers on MTurk for Summarization

Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics Volume 1: Long Papers, pages 14944–14982 July 9-14, 2023 ©2023 Association for Computational Linguistics A Needle in a Haystack: An Analysis of High-Agreement Workers on MTurk for Summarization Lining Zhang,1* Simon Mille,2 Yufang Hou,3 Daniel Deutsch,4 Elizabeth Clark,5 Yixin Liu,6 Saad Mahamood,7 Sebastian Gehrmann,5 Miruna Clinciu,8 Khyathi Chandu,9 João Sedoc1 1New York University, 2ADAPT Centre, DCU, 3IBM Research, 4Google, 5Google Research, 6Yale University, 7trivago N.V., 8University of Edinburgh, 9Allen Institute for AI Abstract To prevent the costly and inefficient use of re- sources on low-quality annotations, we want a method for creating a pool of dependable an- notators who can effectively complete difficult tasks, such as evaluating automatic summariza- tion. Thus, we investigate the recruitment of high-quality Amazon Mechanical Turk workers via a two-step pipeline. We show that we can successfully filter out subpar workers before they carry out the evaluations and obtain high- agreement annotations with similar constraints on resources. Although our workers demon- strate a strong consensus among themselves and CloudResearch workers, their alignment with expert judgments on a subset of the data is not as expected and needs further training in correctness. This paper still serves as a best practice for the recruitment of qualified annota- tors in other challenging annotation tasks. 1 Introduction Natural language generation (NLG) tasks like text summarization are challenging to evaluate both in terms of automatic metrics and human evaluations (Gehrmann et al., 2022). Although automatic met- rics are inexpensive proxies for human annotations for tasks like dialog evaluation (Mehri et al., 2022), they may have problems dealing with paraphrases, capturing distant dependencies, or identifying nu- ances in human languages (Banerjee and Lavie, 2005; Isozaki et al., 2010; Manning et al., 2020). Thus, it is still crucial to obtain high-quality human annotations as gold labels for evaluation. Amazon Mechanical Turk (MTurk)1 is a commonly used crowdsourcing platform for collecting human an- notations on designed tasks, known as Human Intel- ligence Tasks (HITs). However, finding qualified workers for high-quality annotations with a bet- ter inter-annotator agreement (IAA) is challenging, * Correspondence to lz2332@nyu.edu 1https://www.mturk.com/ Figure 1: Two-step pipeline for finding high-agreement MTurk workers: participants who satisfy basic qualifi- cation settings and answer designed questions correctly (Qualification) are subsequently filtered in a longer task (Endurance). The maintained worker list is tested for the true annotation task later (Reference-based). especially for difficult tasks such as text summa- rization. Best practices for recruiting high-quality workers are also poorly understood, and the rela- tionship between high quality and high agreement needs further investigation. To tackle the above issues, we design a recruit- ment pipeline to identify workers who are able to produce high-agreement annotations for the eval- uation of text summarization on MTurk. It com- prises a qualification task and an endurance task, followed by a reference-based task (see Figure 1). In the qualification task, workers who meet pre- defined qualification settings receive instructions and qualification questions, including an attention check (Oppenheimer et al., 2009). …

Content truncated to 500 words