Statistical Confidence Measures for Probabilistic Parsing

International Conference RANLP 2009 - Borovets, Bulgaria, pages 388–392 Statistical Confidence Measures for Probabilistic Parsing Ricardo Sánchez-Sáez, Joan-Andreu Sánchez and José-Miguel Benedí Instituto Tecnológico de Informática Universidad Politécnica de Valencia Camí de Vera s/n, Valencia 46022 (Spain) {rsanchez, jandreu, jbenedi}@dsic.upv.es Abstract We introduce a formal framework that allows the cal- culation of new purely statistical confidence measures for parsing, which are estimated from posterior proba- bility of constituents. These measures allow us to mark each constituent of a parse tree as correct or incor- rect. Experimental assessment using the Penn Tree- bank shows favorable results for the classical confi- dence evaluation metrics: the CER and the ROC curve. We also present preliminar experiments on application of confidence measures to improve parse trees by au- tomatic constituent relabeling. 1 Introduction Many parsing methods exist in the literature, includ- ing those based on Probabilistic Context-Free Grammars (PCFGs). Great effort has been undertaken to improve per- formance of these parsers. First, lexicalization of gram- mars with elaborate smoothing accomplished very promis- ing results [4, 5]. Then, manual tree annotation and non- terminal spliting greatly shortened the gap between un- lexicalized models and their better performing lexicalized counterparts [10, 12]. Later, automatic tree annotation systems, using a nonterminal split-and-merge approach and a hierarchy of progressively refined grammars, pro- vided superior results over the best lexicalized approaches [14, 16, 17]. Last but not least, the most impressive results were achieved by reranking systems, as shown in the semi- supervised method of [15], or the forest reranking approach of [9] which uses packed parse forests (compact structures that contain many possible tree derivations). Given the difficulty and importance of parsing in all of its applications [13], there exists an increasing necessity to detect erroneous syntactic structures therein. This need is even more present in parse trees that are obtained using cur- rent high performing systems, especially if error-free trees are desired. In such a case, the few remaining erroneous parts need to be quickly detected and manually corrected (possibly using interactive methods). Assessing the cor- rectness of the different parts of the parsing is needed for the construction of efficient computer-assisted interactive predictive parsing systems, which will be useful in the cre- ation of new gold standard treebanks [6]. This paper is a step forward in introducing Confidence Measures into the parsing world. Confidence measures are a powerful formalism that have been used to detect individual erroneous words in Auto- matic Speech Recognition (ASR) and Statistical Machine Translation (SMT) output sentences [19, 18]. Once these errors are detected and marked, they can be more easily corrected, either by automatic or manual methods. In parsing, confidence measures detect erroneous con- stituents. Some confidence measures for parsing in the form of combinations of characteristics calculated from n- best lists were proposed in [2]. In our work, we present an alternative more akin to word graph-based methods in ASR and SMT. Other works have proposed to improve parsing results by defining parsing algorithms that try to maximize alter- native …

Content truncated to 500 words