Focus Annotation of Task-based Data: A Comparison of Expert and Crowd-Sourced Annotation in a Reading Comprehension Corpus

Focus Annotation of Task-based Data: A Comparison of Expert and Crowd-Sourced Annotation in a Reading Comprehension Corpus Kordula De Kuthy, Ramon Ziai, Detmar Meurers Sonderforschungsbereich 833 Eberhard Karls Universit¨at T¨ubingen {kdk,rziai,dm}@sfs.uni-tuebingen.de Abstract While the formal pragmatic concepts in information structure, such as the focus of an utterance, are precisely defined in theoretical linguistics and potentially very useful in conceptual and practical terms, it has turned out to be difficult to reliably annotate such notions in corpus data (Ritz et al., 2008; Calhoun et al., 2010). We present a large-scale focus annotation effort designed to overcome this problem. Our annotation study is based on the tasked-based corpus CREG (Ott et al., 2012), which consists of answers to explicitly given reading comprehension questions. We compare focus annotation by trained annotators with a crowd-sourcing setup making use of untrained native speakers. Given the task context and an annotation process incrementally making the question form and answer type explicit, the trained annotators reach substantial agreement for focus annotation. Interestingly, the crowd-sourcing setup also supports high-quality annotation – for specific subtypes of data. Finally, we turn to the question whether the relevance of focus annotation can be extrinsically evaluated. We show that automatic short-answer assessment significantly improves for focus annotated data. The focus annotated CREG corpus is freely available and constitutes the largest such resource for German. Keywords: information structure, focus, crowd sourcing, learner corpora, short-answer assessment 1. Introduction The information structure of a sentence is receiving signif- icant interest in linguistics as the attention has shifted from individual sentences to the question how the information is packaged in sentences analyzed in context. Complement- ing the theoretical interest, identifying information struc- tural concepts has also been shown to be relevant in prac- tical computational linguistic tasks such as Short Answer Assessment: While some approaches have integrated as- pects of givenness (Bailey and Meurers, 2008; Mohler et al., 2011), more recent work (Meurers et al., 2011; Hahn and Meurers, 2012) has argued for relying on focus as dis- cussed in formal pragmatics (e.g., Krifka, 2007, p. 18). In this paper, we present a comprehensive focus annota- tion study based on the tasked-based corpus CREG (Ott et al., 2012), consisting of answers to explicitly given reading comprehension questions. We compare focus annotation by trained annotators with a crowd-sourcing setup making use of untrained native speakers. As a result of these annota- tion efforts, with this paper we provide both a substantial new corpus resource with gold standard focus annotation and conceptual insights into the nature of focus annotation in different types of contexts. While theoretical linguists have discussed the notion of fo- cus for decades (cf., e.g., Jackendoff, 1972; Stechow, 1981; Rooth, 1992; Schwarzschild, 1999; B¨uring, 2007), only few attempts at systematically identifying focus in authen- tic data have been made (e.g., Ritz et al., 2008; Calhoun et al., 2010) . These approaches generally were only rewarded with limited success, as they have tried to identify focus in newspaper text or other data types where no explicit …

Content truncated to 500 words