An Evaluation of Parser Robustness for Ungrammatical Sentences

Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 1765–1774, Austin, Texas, November 1-5, 2016. c⃝2016 Association for Computational Linguistics An Evaluation of Parser Robustness for Ungrammatical Sentences Homa B. Hashemi Intelligent Systems Program University of Pittsburgh hashemi@cs.pitt.edu Rebecca Hwa Computer Science Department University of Pittsburgh hwa@cs.pitt.edu Abstract For many NLP applications that require a parser, the sentences of interest may not be well-formed. If the parser can overlook prob- lems such as grammar mistakes and produce a parse tree that closely resembles the correct analysis for the intended sentence, we say that the parser is robust. This paper compares the performances of eight state-of-the-art depen- dency parsers on two domains of ungrammat- ical sentences: learner English and machine translation outputs. We have developed an evaluation metric and conducted a suite of ex- periments. Our analyses may help practition- ers to choose an appropriate parser for their tasks, and help developers to improve parser robustness against ungrammatical sentences. 1 Introduction Previous works have shown that, in general, parser performances degrade when applied to out-of- domain sentences (Gildea, 2001; McClosky et al., 2010; Foster, 2010; Petrov et al., 2010; Foster et al., 2011). If a parser performs reasonably well for a wide range of out-of-domain sentences, it is said to be robust (Bigert et al., 2005; Kakkonen, 2007; Fos- ter, 2007). Sentences that are ungrammatical, awkward, or too casual/colloquial can all be seen as special kinds of out-of-domain sentences. These types of sen- tences are commonplace for NLP applications, from product reviews and social media analysis to intel- ligent language tutors and multilingual processing. Since parsing is an essential component for many applications, it is natural to ask: Are some parsers more robust than others against sentences that are not well-formed? Previous works on parser evalu- ation that focused on accuracy and speed (Choi et al., 2015; Kummerfeld et al., 2012; McDonald and Nivre, 2011; Kong and Smith, 2014) have not taken ungrammatical sentences into consideration. In this paper, we report a set of empirical analy- ses of eight leading dependency parsers on two do- mains of ungrammatical text: English-as-a-Second Language (ESL) learner text and machine transla- tion (MT) outputs. We also vary the types of train- ing sources; the parsers are trained with the Penn Treebank (to be comparable with other studies) and Tweebank, a treebank on tweets (to be a bit more like the test domain) (Kong et al., 2014). The main contributions of the paper are: • a metric and methodology for evaluating un- grammatical sentences without referring to a gold standard corpus; • a quantitative comparison of parser accuracy of leading dependency parsers on ungrammatical sentences; this may help practitioners to select an appropriate parser for their applications; and • a suite of robustness analyses for the parsers on specific kinds of problems in the ungrammati- cal sentences; this may help developers to im- prove parser robustness in the future. 2 Evaluation of Parser Robustness Parser evaluation for ungrammatical texts presents some domain-specific …

Content truncated to 500 words