Solving the Problem of Cascading Errors: Approximate Bayesian Inference for Linguistic Annotation Pipelines

Proceedings of the 2006 Conference on Empirical Methods in Natural Language Processing (EMNLP 2006), pages 618–626, Sydney, July 2006. c⃝2006 Association for Computational Linguistics Solving the Problem of Cascading Errors: Approximate Bayesian Inference for Linguistic Annotation Pipelines Jenny Rose Finkel, Christopher D. Manning and Andrew Y. Ng Computer Science Department Stanford University Stanford, CA 94305 {jrfinkel, manning, ang}@cs.stanford.edu Abstract The end-to-end performance of natural language processing systems for com- pound tasks, such as question answering and textual entailment, is often hampered by use of a greedy 1-best pipeline archi- tecture, which causes errors to propagate and compound at each stage. We present a novel architecture, which models these pipelines as Bayesian networks, with each low level task corresponding to a variable in the network, and then we perform ap- proximate inference to find the best la- beling. Our approach is extremely sim- ple to apply but gains the benefits of sam- pling the entire distribution over labels at each stage in the pipeline. We apply our method to two tasks – semantic role la- beling and recognizing textual entailment – and achieve useful performance gains from the superior pipeline architecture. 1 Introduction Almost any system for natural language under- standing must recover hidden linguistic structure at many different levels: parts of speech, syntac- tic dependencies, named entities, etc. For exam- ple, modern semantic role labeling (SRL) systems use the parse of the sentence, and question answer- ing requires question type classification, parsing, named entity tagging, semantic role labeling, and often other tasks, many of which are dependent on one another and must be pipelined together. Pipelined systems are ubiquitous in NLP: in ad- dition to the above examples, commonly parsers and named entity recognizers use part of speech tags and chunking information, and also word seg- mentation for languages such as Chinese. Almost no NLP task is truly standalone. Most current systems for higher-level, aggre- gate NLP tasks employ a simple 1-best feed for- ward architecture: they greedily take the best out- put at each stage in the pipeline and pass it on to the next stage. This is the simplest architecture to build (particularly if reusing existing component systems), but errors are frequently made during this pipeline of annotations, and when a system is given incorrectly labeled input it is much harder for that system to do its task correctly. For ex- ample, when doing semantic role labeling, if no syntactic constituent of the parse actually corre- sponds to a given semantic role, then that seman- tic role will almost certainly be misidentified. It is therefore disappointing, but not surprising, that F-measures on SRL drop more than 10% when switching from gold parses to automatic parses (for instance, from 91.2 to 80.0 for the joint model of Toutanova (2005)). A common improvement on this architecture is to pass k-best lists between processing stages, for example (Sutton and McCallum, 2005; Wellner et al., 2004). Passing on a k-best list gives useful improvements (e.g., in Koomen et al. (2005)), but efficiently …

Content truncated to 500 words