Parser Features for Sentence Grammaticality Classification

Parser Features for Sentence Grammaticality Classification Sze-Meng Jojo Wong Centre for Language Technology Macquarie University Sydney, NSW, Australia sze.wong@mq.edu.au Mark Dras Centre for Language Technology Macquarie University Sydney, NSW, Australia mark.dras@mq.edu.au Abstract Automatically judging sentences for their grammaticality is potentially useful for several purposes — evaluating language technology systems, assessing language competence of second or foreign lan- guage learners, and so on. Previous work has examined parser ‘byproducts’, in par- ticular parse probabilities, to distinguish grammatical sentences from ungrammat- ical ones. The aim of the present paper is to examine whether the primary out- put of a parser, which we characterise via CFG production rules embodied in a parse, contains useful information for sentence grammaticality classification; and also to examine which feature selection metrics are most useful in this task. Our re- sults show that using gold standard pro- duction rules alone can improve over us- ing parse probabilities alone. Combin- ing parser-produced production rules with parse probabilities further produces an im- provement of 1.6% on average in the over- all classification accuracy. 1 Introduction Automatically judging sentences for their gram- maticality has been a long-standing research prob- lem within the natural language processing com- munity. The ability of distinguishing grammati- cal sentences from ungrammatical ones has many potential applications, which include evaluating language technology systems such as natural lan- guage generation (Mutton et al., 2007) and ma- chine translation (Gamon et al., 2005), as well as assessing language competence of second lan- guage or foreign language learners (Brockett et al., 2006; Gamon et al., 2008; Han et al., 2010). Various approaches have been proposed in the past to address this typical classification problem. A number of these existing studies attempt to ex- ploit some form of ‘parser byproduct’ as classifi- cation features for machine learning: for instance, (log) probability of a parse tree, number of par- tial (incomplete) parse trees, parsing duration, and such (Mutton et al., 2007; Sun et al., 2007; Fos- ter et al., 2008; Wagner et al., 2009). The aim of this paper is to examine whether the primary out- put of a parser contains useful information for this classification problem; we characterise this infor- mation by the CFG production rules embodied in a parse. The intuition is that particular production rules might be strongly characteristic of ungram- matical sentences, and that looking at individual rules might provide clues that are aggregated out in measures such as the parse tree probability. We carry out experiments to test this intuition, using the Penn treebank and an artificially cre- ated ungrammatical version created by Foster et al. (2008). This allows a large amount of data to be used for classification, embodying a controlled ungrammaticality that is suitable for preliminary work in this direction; it is for similar reasons that construction of erroneous corpora has become a more prominent line of computational linguistic research lately (Foster and Andersen, 2009; Han et al., 2010; Dickinson, 2010). The present study is carried out in two stages. In the first stage, following …

Content truncated to 500 words