Problems in Evaluating Grammatical Error Detection Systems

Proceedings of COLING 2012: Technical Papers, pages 611–628, COLING 2012, Mumbai, December 2012. Problems in Evaluating Grammatical Error Detection Systems Martin CHODOROW 1 Markus DICKINSON 2 Ross ISRAEL2 Joel T ETREAU LT 3 (1) Hunter College and the Graduate Center, City University of New York, New York, NY 10065, USA (2) Indiana University, Department of Linguistics, Bloomington, IN 47405, USA (3) Educational Testing Service, Rosedale Road, Princeton, NJ 08540, USA mchodoro@hunter.cuny.edu, md7@indiana.edu, raisrael@indiana.edu, jtetreault@ets.org ABSTRACT Many evaluation issues for grammatical error detection have previously been overlooked, making it hard to draw meaningful comparisons between different approaches, even when they are evaluated on the same corpus. To begin with, the three-way contingency between a writer’s sentence, the annotator’s correction, and the system’s output makes evaluation more complex than in some other NLP tasks, which we address by presenting an intuitive evaluation scheme. Of particular importance to error detection is the skew of the data – the low frequency of errors as compared to non-errors – which distorts some traditional measures of performance and limits their usefulness, leading us to recommend the reporting of raw measurements (true positives, false negatives, false positives, true negatives). Other issues that are particularly vexing for error detection focus on defining these raw measurements: specifying the size or scope of an error, properly treating errors as graded rather than discrete phenomena, and counting non-errors. We discuss recommendations for best practices with regard to reporting the results of system evaluation for these cases, recommendations which depend upon making clear one’s assumptions and applications for error detection. By highlighting the problems with current error detection evaluation, the field will be better able to move forward. KEYWORDS: grammatical error detection, system evaluation, evaluation metrics. 611 1 Introduction With hundreds of millions of people worldwide learning second, or even third, languages (Leacock et al., 2010), there is a large and growing need for NLP systems that automatically detect and correct the grammar and word usage errors that learners make. In response to this need, NLP researchers have developed tools to target errors involving articles (Han et al., 2006), prepositions (Tetreault and Chodorow, 2008b), particles (Dickinson et al., 2011), verb forms (Lee and Seneff, 2008), and collocations (Dahlmeier and Ng, 2011). Research in this field has surged over the last few years, culminating notably in two recent “Helping Our Own” (HOO) Shared Tasks (Dale and Kilgarriff, 2010) and (Dale et al., 2012), which are concerned with automated error correction in texts authored by non-native speakers of English working in NLP. However, despite this high level of activity and interest, there is relatively little consensus on how best to evaluate grammatical error detection/correction systems. This makes it hard to measure performance and compare systems, which is essential for progress in the field. The goal of this paper is to draw attention to the many evaluation issues in error detection that have largely been overlooked in the past and which make it hard to draw meaningful comparisons between different approaches, even when they are …

Content truncated to 500 words