Posterior calibration and exploratory analysis for natural language processing models

Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pages 1587–1598, Lisbon, Portugal, 17-21 September 2015. c⃝2015 Association for Computational Linguistics. Posterior calibration and exploratory analysis for natural language processing models Khanh Nguyen Department of Computer Science University of Maryland, College Park College Park, MD 20742 kxnguyen@cs.umd.edu Brendan O’Connor College of Information and Computer Sciences University of Massachusetts, Amherst Amherst, MA, 01003 brenocon@cs.umass.edu Abstract Many models in natural language process- ing define probabilistic distributions over linguistic structures. We argue that (1) the quality of a model’s posterior distribu- tion can and should be directly evaluated, as to whether probabilities correspond to empirical frequencies; and (2) NLP uncer- tainty can be projected not only to pipeline components, but also to exploratory data analysis, telling a user when to trust and not trust the NLP analysis. We present a method to analyze calibration, and apply it to compare the miscalibration of sev- eral commonly used models. We also con- tribute a coreference sampling algorithm that can create confidence intervals for a political event extraction task.1 1 Introduction Natural language processing systems are imper- fect. Decades of research have yielded analyzers that mis-identify named entities, mis-attach syn- tactic relations, and mis-recognize noun phrase coreference anywhere from 10-40% of the time. But these systems are accurate enough so that their outputs can be used as soft, if noisy, indicators of language meaning for use in downstream analysis, such as systems that perform question answering, machine translation, event extraction, and narra- tive analysis (McCord et al., 2012; Gimpel and Smith, 2008; Miwa et al., 2010; Bamman et al., 2013). To understand the performance of an ana- lyzer, researchers and practitioners typically mea- sure the accuracy of individual labels or edges among a single predicted output structure y, such as a most-probable tagging or entity clustering arg maxy P(y|x) (conditional on text data x). 1See the extended version of this paper for software, ap- pendix, and acknowledgments sections: http://brenocon.com/nlpcalib/ http://arxiv.org/abs/1508.05154 But a probabilistic model gives a probability distribution over many other output structures that have smaller predicted probabilities; a line of work has sought to control cascading pipeline errors by passing on multiple structures from earlier stages of analysis, by propagating prediction uncertainty through multiple samples (Finkel et al., 2006), K-best lists (Venugopal et al., 2008; Toutanova et al., 2008), or explicitly diverse lists (Gimpel et al., 2013); often the goal is to marginalize over structures to calculate and minimize an expected loss function, as in minimum Bayes risk decod- ing (Goodman, 1996; Kumar and Byrne, 2004), or to perform joint inference between early and later stages of NLP analysis (e.g. Singh et al., 2013; Durrett and Klein, 2014). These approaches should work better when the posterior probabilities of the predicted linguistic structures reflect actual probabilities of the struc- tures or aspects of the structures. For example, say a model is overconfident: it places too much prob- ability mass in the top prediction, and not enough in the rest. Then there will be …

Content truncated to 500 words