Charting Democracy Across Parsers

Charting Democracy Across Parsers Scott Nowson1 and Robert Dale Centre for Language Technology Macquarie University Sydney, Australia {snowson|rdale}@ics.mq.edu.au Abstract Different parsers trained on the same cor- pus deliver different results, both in terms of overall performance and in terms of the individual analyses they provide. In par- ticular, for any given sentence, one parser may provide a correct analysis, while an- other will produce an incorrect analysis; but when faced with a different sentence, the first parser may be in error while the sec- ond is correct. In this paper, we leverage this observation by exploring how the re- sults of a number of different parsers may be combined to provide a better performance than any single parser. The method involves constructing a chart that contains edges con- tributed by a collection of parsers, with a simple voting mechanism to choose the most preferred constituents; this provides a signif- icant improvement in performance over any individual parser. More sophisticated voting mechanisms are also discussed. 1 Introduction Parsers make mistakes. This is perhaps most appar- ent when a parser trained on a given corpus is ap- plied to data from a domain or genre different to that of the training corpus. One can, of course, retrain the parser on new data that is more representative of the texts to be handled; but annotation is an expen- sive process, and the literature does not provide a great deal of guidance as to how much annotation is 1Scott Nowson is now at Appen Pty Ltd. required in order to obtain an acceptable result (but see Reichart and Rappoport (2007a) for some recent interesting results in this area). Unfortunately, parsers make mistakes even on the corpora on which they are trained. Before we begin to consider how we might adapt a parser to a new domain, we are therefore interested in how we might improve the performance of existing parsers on the corpora used to derive their models. We make the observation that different parsers have different ‘error profiles’, by which we mean that different parsers do not necessarily make the same mistakes. Consider the following verb phrase taken from our test corpus: ...lock in profits by buying futures when futures prices fall Figure 1 shows the analyses provided for this verb phrase by three different parsers, as an illustration of the kinds of disagreements that are common. In the first analysis, in is misclassified as a preposition, while in the second and third analyses it is correctly analysed as a particle. However, the second parse contains a misparse of the embedded VP buying fu- tures, while this is correctly analysed in the first and third parses. This leads us to the hypothesis that, if we were able to select for each parser those parts of individ- ual parses that are more likely to be correct, then the overall result would be an improvement upon the analysis of any individual parser. We explore this hypothesis in this paper, by providing a framework within which …

Content truncated to 500 words