A Comparative Evaluation of Deep and Shallow Approaches to the Automatic Detection of Common Grammatical Errors

Proceedings of the 2007 Joint Conference on Empirical Methods in Natural Language Processing and Computational Natural Language Learning, pp. 112–121, Prague, June 2007. c⃝2007 Association for Computational Linguistics A Comparative Evaluation of Deep and Shallow Approaches to the Automatic Detection of Common Grammatical Errors Joachim Wagner, Jennifer Foster, and Josef van Genabith∗ National Centre for Language Technology School of Computing, Dublin City University, Dublin 9, Ireland {jwagner, jfoster, josef}@computing.dcu.ie Abstract This paper compares a deep and a shallow processing approach to the problem of clas- sifying a sentence as grammatically well- formed or ill-formed. The deep processing approach uses the XLE LFG parser and En- glish grammar: two versions are presented, one which uses the XLE directly to perform the classification, and another one which uses a decision tree trained on features con- sisting of the XLE’s output statistics. The shallow processing approach predicts gram- maticality based on n-gram frequency statis- tics: we present two versions, one which uses frequency thresholds and one which uses a decision tree trained on the frequen- cies of the rarest n-grams in the input sen- tence. We find that the use of a decision tree improves on the basic approach only for the deep parser-based approach. We also show that combining both the shallow and deep decision tree features is effective. Our eval- uation is carried out using a large test set of grammatical and ungrammatical sentences. The ungrammatical test set is generated au- tomatically by inserting grammatical errors into well-formed BNC sentences. 1 Introduction This paper is concerned with the task of predict- ing whether a sentence contains a grammatical er- ror. An accurate method for carrying out automatic ∗Also affiliated to IBM CAS, Dublin. grammaticality judgements has uses in the areas of computer-assisted language learning and grammar checking. Comparative evaluation of existing error detection approaches has been hampered by a lack of large and commonly used evaluation error cor- pora. We attempt to overcome this by automatically creating a large error corpus, containing four dif- ferent types of frequently occurring grammatical er- rors. We use this corpus to evaluate the performance of two approaches to the task of automatic error de- tection. One approach uses low-level detection tech- niques based on POS n-grams. The other approach is a novel parser-based method which employs deep linguistic processing to discriminate grammatical in- put from ungrammatical. For both approaches, we implement a basic solution, and then attempt to im- prove upon this solution using a decision tree clas- sifier. We show that combining both methods im- proves upon the individual methods. N-gram-based approaches to the problem of error detection have been proposed and implemented in various forms by Atwell(1987), Bigert and Knutsson (2002), and Chodorow and Leacock (2000) amongst others. Existing approaches are hard to compare since they are evaluated on different test sets which vary in size and error density. Furthermore, most of these approaches concentrate on one type of gram- matical error only, namely, context-sensitive or real- word spelling errors. We implement a …

Content truncated to 500 words