Crowd Prefers the Middle Path: A New IAA Metric for Crowdsourcing Reveals Turker Biases in Query Segmentation

Proceedings of the 51st Annual Meeting of the Association for Computational Linguistics, pages 1713–1722, Sofia, Bulgaria, August 4-9 2013. c⃝2013 Association for Computational Linguistics Crowd Prefers the Middle Path: A New IAA Metric for Crowdsourcing Reveals Turker Biases in Query Segmentation Rohan Ramanath∗ R. V. College of Engineering Bangalore, India ronramanath@gmail.com Monojit Choudhury Microsoft Research Lab India Bangalore, India monojitc@microsoft.com Kalika Bali Microsoft Research Lab India Bangalore, India kalikab@microsoft.com Rishiraj Saha Roy† Indian Institute of Technology Kharagpur Kharagpur, India rishiraj@cse.iitkgp.ernet.in Abstract Query segmentation, like text chunking, is the first step towards query understand- ing. In this study, we explore the effec- tiveness of crowdsourcing for this task. Through carefully designed control ex- periments and Inter Annotator Agreement metrics for analysis of experimental data, we show that crowdsourcing may not be a suitable approach for query segmentation because the crowd seems to have a very strong bias towards dividing the query into roughly equal (often only two) parts. Sim- ilarly, in the case of hierarchical or nested segmentation, turkers have a strong prefer- ence towards balanced binary trees. 1 Introduction Text chunking of Natural Language (NL) sentences is a well studied problem that is an essential pre- processing step for many NLP applications (Ab- ney, 1991; Abney, 1995). In the context of Web search queries, query segmentation is similarly the first step towards analysis and understanding of queries (Hagen et al., 2011). The task in both the cases is to divide the sentence or the query into contiguous segments or chunks of words such that the words from a segment are related to each other more strongly than words from different segments (Bendersky et al., 2009). It is typically assumed that the segments are structurally and semantically coherent and, therefore, the information contained in them can be processed holistically. ∗The work was done during author’s internship at Mi- crosoft Research Lab India. † This author was supported by Microsoft Corporation and Microsoft Research India under the Microsoft Research India PhD Fellowship Award. f Pipe representation Boundary var. 4 apply | first aid course | on line 1 0 0 1 0 3 apply first aid course | on line 0 0 0 1 0 2 apply first aid | course on line 0 0 1 0 0 1 apply | first aid | course | on line 1 0 1 1 0 Table 1: Example of flat segmentation by Turkers. f is the frequency of annotations; segment bound- aries are represented by |. f Bracket representation Boundary var. 4 ((apply first) ((aid course) (on line))) 0 2 0 1 0 2 (((apply (first aid)) course) (on line)) 1 0 2 3 0 2 ((apply ((first aid) course)) (on line)) 2 0 1 3 0 1 (apply (((first aid) course) (on line))) 3 0 1 2 0 1 ((apply (first aid)) (course (on line))) 1 0 2 1 0 Table 2: Example of nested segmentation by Turk- ers. f is the frequency of annotations. A majority of work on query segmentation re- …

Content truncated to 500 words