Creating a manually error-tagged and shallow-parsed learner corpus

Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics, pages 1210–1219, Portland, Oregon, June 19-24, 2011. c⃝2011 Association for Computational Linguistics Creating a manually error-tagged and shallow-parsed learner corpus Ryo Nagata Konan University 8-9-1 Okamoto, Kobe 658-0072 Japan rnagata @ konan-u.ac.jp. Edward Whittaker Vera Sheinman The Japan Institute for Educational Measurement Inc. 3-2-4 Kita-Aoyama, Tokyo, 107-0061 Japan � whittaker,sheinman � @jiem.co.jp Abstract The availability of learner corpora, especially those which have been manually error-tagged or shallow-parsed, is still limited. This means that researchers do not have a common devel- opment and test set for natural language pro- cessing of learner English such as for gram- matical error detection. Given this back- ground, we created a novel learner corpus that was manually error-tagged and shallow- parsed. This corpus is available for research and educational purposes on the web. In this paper, we describe it in detail together with its data-collection method and annota- tion schemes. Another contribution of this paper is that we take the first step toward evaluating the performance of existing POS- tagging/chunking techniques on learner cor- pora using the created corpus. These contribu- tions will facilitate further research in related areas such as grammatical error detection and automated essay scoring. 1 Introduction The availability of learner corpora is still somewhat limited despite the obvious usefulness of such data in conducting research on natural language process- ing of learner English in recent years. In particular, learner corpora tagged with grammatical errors are rare because of the difficulties inherent in learner corpus creation as will be described in Sect. 2. As shown in Table 1, error-tagged learner corpora are very few among existing learner corpora (see Lea- cock et al. (2010) for a more detailed discussion of learner corpora). Even if data is error-tagged, it is often not available to the public or its access is severely restricted. For example, the Cambridge Learner Corpus, which is one of the largest error- tagged learner corpora, can only be used by authors and writers working for Cambridge University Press and by members of staff at Cambridge ESOL. Error-tagged learner corpora are crucial for devel- oping and evaluating error detection/correction al- gorithms such as those described in (Rozovskaya and Roth, 2010b; Chodorow and Leacock, 2000; Chodorow et al., 2007; Felice and Pulman, 2008; Han et al., 2004; Han et al., 2006; Izumi et al., 2003b; Lee and Seneff, 2008; Nagata et al., 2004; Nagata et al., 2005; Nagata et al., 2006; Tetreault et al., 2010b). This is one of the most active research areas in natural language processing of learner En- glish. Because of the restrictions on their availabil- ity, researchers have used their own learner corpora to develop and evaluate error detection/correction methods, which are often not commonly available to other researchers. This means that the detec- tion/correction performance of each existing method is not directly comparable as Rozovskaya and Roth (2010a) and Tetreault et al. (2010a) point out. In other words, we are not sure which methods achieve …

Content truncated to 500 words