Error Mining in Parsing Results

Proceedings of the 21st International Conference on Computational Linguistics and 44th Annual Meeting of the ACL, pages 329–336, Sydney, July 2006. c⃝2006 Association for Computational Linguistics Error mining in parsing results Benoît Sagot and Éric de la Clergerie Projet ATOLL - INRIA Domaine de Voluceau, B.P. 105 78153 Le Chesnay Cedex, France {benoit.sagot,eric.de_la_clergerie}@inria.fr Abstract We introduce an error mining technique for automatically detecting errors in re- sources that are used in parsing systems. We applied this technique on parsing re- sults produced on several million words by two distinct parsing systems, which share the syntactic lexicon and the pre-parsing processing chain. We were thus able to identify missing and erroneous informa- tion in these resources. 1 Introduction Natural language parsing is a hard task, partly be- cause of the complexity and the volume of infor- mation that have to be taken into account about words and syntactic constructions. However, it is necessary to have access to such information, stored in resources such as lexica and grammars, and to try and minimize the amount of missing and erroneous information in these resources. To achieve this, the use of these resources at a large- scale in parsers is a very promising approach (van Noord, 2004), and in particular the analysis of sit- uations that lead to a parsing failure: one can learn from one’s own mistakes. We introduce a probabilistic model that allows to identify forms and form bigrams that may be the source of errors, thanks to a corpus of parsed sentences. In order to facilitate the exploitation of forms and form bigrams detected by the model, and in particular to identify causes of errors, we have developed a visualization environment. The whole system has been tested on parsing results produced for several multi-million-word corpora and with two different parsers for French, namely SXLFG and FRMG. However, the error mining technique which is the topic of this paper is fully system- and language-independent. It could be applied with- out any change on parsing results produced by any system working on any language. The only infor- mation that is needed is a boolean value for each sentence which indicates if it has been success- fully parsed or not. 2 Principles 2.1 General idea The idea we implemented is inspired from (van Noord, 2004). In order to identify missing and er- roneous information in a parsing system, one can analyze a large corpus and study with statistical tools what differentiates sentences for which pars- ing succeeded from sentences for which it failed. The simplest application of this idea is to look for forms, called suspicious forms, that are found more frequently in sentences that could not be parsed. This is what van Noord (2004) does, with- out trying to identify a suspicious form in any sen- tence whose parsing failed, and thus without tak- ing into account the fact that there is (at least) one cause of error in each unparsable sentence.1 On the contrary, we will look, in each sentence on which …

Content truncated to 500 words