Annotating Article Errors in Spanish Learner Texts: Design and Evaluation of an Annotation Scheme

PACLIC 28 !234 Annotating article errors in Spanish learner texts: design and evaluation of an annotation scheme Mar´ıa del Pilar Valverde Iba˜nez Faculty of Foreign Languages Aichi Prefectural University 1522-3 Ibaragasama, Nagakute-shi Aichi, 480-1198, Japan valverde@for.aichi-pu.ac.jp Akira Ohtani Faculty of Informatics Osaka Gakuin University 2-36-1 Kishibe-minami, Suita-shi Osaka, 564-8511, Japan ohtani@ogu.ac.jp Abstract Annotating a corpus with error information is a challenging task. This paper describes the design, evaluation and refinement of an an- notation scheme for Spanish article errors in learner data, so that future work on corpus an- notation and automatic article error detection can progress. To evaluate reliability, 300 noun phrases with definite, indefinite and zero arti- cle have been tagged by four annotators. We analysed different types of disagreement, pre- sented suggestions to increase reliability and applied the refined annotation scheme to cre- ate a gold-standard annotation. 1 Introduction The annotation of learner texts with error informa- tion is necessary for linguistic research as well as for the development of language learning applica- tions using natural language processing (NLP) tech- niques. While much efforts have concentrated on English, it is necessary to develop learner corpora and tools for other foreign languages like Spanish. This is the most commonly studied foreign language in the United States and the second most studied for- eign language -after English- in many other coun- tries. Overall, it is estimated that nearly 20 million people are studying Spanish as a foreign language (Instituto Cervantes, 2013). However, learner cor- pora and tools for this language are scarce (Lozano and Mendikoetxea, 2013; Nazar and Renau, 2012; del Pilar Valverde and Ohtani, 2012; Wanner et al., 2013). The goal of this paper is to define an annota- tion scheme that is suitable for reliable Spanish ar- ticle error annotation, so that future work on corpus annotation and automatic article error detection can progress. Automatic detection of errors has focused on function words such as articles (Izumi et al., 2004; Han et al., 2006; Felice and Pulman, 2008b; Gamon et al., 2008; Yi et al., 2008), prepositions (Felice and Pulman, 2008a) and particles (Dickinson, 2008; Oyama and Matsumoto, 2010). Function words are the most frequent words in any language, and they are also a very common source of mistakes for learn- ers. As for error annotation, one of the main difficul- ties is reliability. For some learner errors, like num- ber and gender agreement, rules are clearly defined. Other kind of errors, like article or preposition pres- ence and choice, are harder to annotate because na- tive speakers differ widely with respect to what is acceptable usage. For article and noun number se- lection, for example, in Lee et al. (2009) raters found more than one valid construction for more than 18% of noun phrases. To address this problem, we experiment with a preliminary annotation scheme for article errors, analyse the form disagreement among annotators takes, and refine the annotation scheme according to it. The paper is organized as follows. In section 2 we briefly summarize the …

Content truncated to 500 words