Learning with Probabilistic Features for Improved Pipeline Models

Proceedings of the 2008 Conference on Empirical Methods in Natural Language Processing, pages 670–679, Honolulu, October 2008. c⃝2008 Association for Computational Linguistics Learning with Probabilistic Features for Improved Pipeline Models Razvan C. Bunescu School of EECS Ohio University Athens, OH 45701 bunescu@ohio.edu Abstract We present a novel learning framework for pipeline models aimed at improving the com- munication between consecutive stages in a pipeline. Our method exploits the confidence scores associated with outputs at any given stage in a pipeline in order to compute prob- abilistic features used at other stages down- stream. We describe a simple method of in- tegrating probabilistic features into the linear scoring functions used by state of the art ma- chine learning algorithms. Experimental eval- uation on dependency parsing and named en- tity recognition demonstrate the superiority of our approach over the baseline pipeline mod- els, especially when upstream stages in the pipeline exhibit low accuracy. 1 Introduction Machine learning algorithms are used extensively in natural language processing. Applications range from fundamental language tasks such as part of speech (POS) tagging or syntactic parsing, to higher level applications such as information extraction (IE), semantic role labeling (SRL), or question an- swering (QA). Learning a model for a particular lan- guage processing problem often requires the output from other natural language tasks. Syntactic pars- ing and dependency parsing usually start with a tex- tual input that is tokenized, split in sentences and POS tagged. In information extraction, named en- tity recognition (NER), coreference resolution, and relation extraction (RE) have been shown to benefit from features that use POS tags and syntactic depen- dencies. Similarly, most SRL approaches assume a parse tree representation of the input sentences. The common practice in modeling such dependen- cies is to use a pipeline organization, in which the output of one task is fed as input to the next task in the sequence. One advantage of this model is that it is very simple to implement; it also allows for a modular approach to natural language process- ing. The key disadvantage is that errors propagate between stages in the pipeline, significantly affect- ing the quality of the final results. One solution is to solve the tasks jointly, using the principled framework of probabilistic graphical models. Sut- ton et al. (2004) use factorial Conditional Random Fields (CRFs) (Lafferty et al., 2001) to jointly pre- dict POS tags and segment noun phrases, improving on the cascaded models that perform the two tasks in sequence. Wellner et al. (2004) describe a CRF model that integrates the tasks of citation segmen- tation and citation matching. Their empirical results show the superiority of the integrated model over the pipeline approach. While more accurate than their pipeline analogues, probabilistic graphical models that jointly solve multiple natural language tasks are generally more demanding in terms of finding the right representations, the associated inference algo- rithms and their computational complexity. Recent negative results on the integration of syntactic pars- ing with SRL (Sutton and McCallum, 2005) provide additional evidence …

Content truncated to 500 words