The Ups and Downs of Preposition Error Detection in ESL Writing

Proceedings of the 22nd International Conference on Computational Linguistics (Coling 2008), pages 865–872 Manchester, August 2008 The Ups and Downs of Preposition Error Detection in ESL Writing Joel R. Tetreault Educational Testing Service 660 Rosedale Road Princeton, NJ, USA JTetreault@ets.org Martin Chodorow Hunter College of CUNY 695 Park Avenue New York, NY, USA martin.chodorow@hunter.cuny.edu Abstract In this paper we describe a methodology for detecting preposition errors in the writ- ing of non-native English speakers. Our system performs at 84% precision and close to 19% recall on a large set of stu- dent essays. In addition, we address the problem of annotation and evaluation in this domain by showing how current ap- proaches of using only one rater can skew system evaluation. We present a sampling approach to circumvent some of the issues that complicate evaluation of error detec- tion systems. 1 Introduction The long-term goal of our work is to develop a system which detects errors in grammar and us- age so that appropriate feedback can be given to non-native English writers, a large and growing segment of the world’s population. Estimates are that in China alone as many as 300 million peo- ple are currently studying English as a second lan- guage (ESL). Usage errors involving prepositions are among the most common types seen in the writing of non-native English speakers. For ex- ample, (Izumi et al., 2003) reported error rates for English prepositions that were as high as 10% in a Japanese learner corpus. Errors can involve in- correct selection (“we arrived to the station”), ex- traneous use (“he went to outside”), and omission (“we are fond null beer”). What is responsible for making preposition usage so difficult for non- native speakers? c⃝ 2008. Licensed under the Creative Commons Attribution-Noncommercial-Share Alike 3.0 Unported li- cense (http://creativecommons.org/licenses/by-nc-sa/3.0/). Some rights reserved. At least part of the difficulty seems to be due to the great variety of linguistic functions that prepo- sitions serve. When a preposition marks the ar- gument of a predicate, such as a verb, an ad- jective, or a noun, preposition selection is con- strained by the argument role that it marks, the noun which fills that role, and the particular predi- cate. Many English verbs also display alternations (Levin, 1993) in which an argument is sometimes marked by a preposition and sometimes not (e.g., “They loaded the wagon with hay” / “They loaded hay on the wagon”). When prepositions introduce adjuncts, such as those of time or manner, selec- tion is constrained by the object of the preposition (“at length”, “in time”, “with haste”). Finally, the selection of a preposition for a given context also depends upon the intended meaning of the writer (“we sat at the beach”, “on the beach”, “near the beach”, “by the beach”). With so many sources of variation in English preposition usage, we wondered if the task of se- lecting a preposition for a given context might prove challenging even for native speakers. To investigate this possibility, we randomly selected 200 sentences …

Content truncated to 500 words