Data Driven Grammatical Error Detection in Transcripts of Children’s Speech

Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 980–989, October 25-29, 2014, Doha, Qatar. c⃝2014 Association for Computational Linguistics Data Driven Grammatical Error Detection in Transcripts of Children’s Speech Eric Morley CSLU OHSU Portland, OR 97239 morleye@gmail.com Anna Eva Hallin Department of Communicative Sciences and Disorders New York University New York, NY ae.hallin@nyu.edu Brian Roark Google Research New York, NY 10011 roarkbr@gmail.com Abstract We investigate grammatical error detec- tion in spoken language, and present a data-driven method to train a dependency parser to automatically identify and label grammatical errors. This method is ag- nostic to the label set used, and the only manual annotations needed for training are grammatical error labels. We find that the proposed system is robust to disfluencies, so that a separate stage to elide disfluen- cies is not required. The proposed system outperforms two baseline systems on two different corpora that use different sets of error tags. It is able to identify utterances with grammatical errors with an F1-score as high as 0.623, as compared to a baseline F1 of 0.350 on the same data. 1 Introduction Research into automatic grammatical error detec- tion has primarily been motivated by the task of providing feedback to writers, whether they be na- tive speakers of a language or second language learners. Grammatical error detection, however, is also useful in the clinical domain, for example, to assess a child’s ability to produce grammatical lan- guage. At present, clinicians and researchers into child language must manually identify and clas- sify particular kinds of grammatical errors in tran- scripts of children’s speech if they wish to assess particular aspects of the child’s linguistic ability from a sample of spoken language. Such manual annotation, which is called language sample anal- ysis in the clinical field, is expensive, hindering its widespread adoption. Manual annotations may also be inconsistent, particularly between different research groups, which may be investigating dif- ferent phenomena. Automated grammatical error detection has the potential to address both of these issues, being both cheap and consistent. Aside from performance, there are at least two key requirements for a grammatical error detector to be useful in a clinical setting: 1) it must be able to handle spoken language, and 2) it must be train- able. Clinical data typically consists of transcripts of spoken language, rather than formal written lan- guage. As a result, a system must be prepared to handle disfluencies, utterance fragments, and other phenomena that are entirely grammatical in speech, but not in writing. On the other hand, a system designed for transcripts of speech does not need to identify errors specific to written language such as punctuation or spelling mistakes. Further- more, a system designed for clinical data must be able to handle language produced by children who may have atypical language due to a developmen- tal disorder, and therefore may produce grammati- cal errors that would be unexpected in written lan- guage. A grammatical error detector appropriate for a clinical setting must …

Content truncated to 500 words