Estimating effect size across datasets

Proceedings of NAACL-HLT 2013, pages 607–611, Atlanta, Georgia, 9–14 June 2013. c⃝2013 Association for Computational Linguistics Estimating effect size across datasets Anders Søgaard Center for Language Technology University of Copenhagen soegaard@hum.ku.dk Abstract Most NLP tools are applied to text that is dif- ferent from the kind of text they were eval- uated on. Common evaluation practice pre- scribes significance testing across data points in available test data, but typically we only have a single test sample. This short paper argues that in order to assess the robustness of NLP tools we need to evaluate them on diverse samples, and we consider the prob- lem of finding the most appropriate way to es- timate the true effect size across datasets of our systems over their baselines. We apply meta-analysis and show experimentally – by comparing estimated error reduction over ob- served error reduction on held-out datasets – that this method is significantly more predic- tive of success than the usual practice of using macro- or micro-averages. Finally, we present a new parametric meta-analysis based on non- standard assumptions that seems superior to standard parametric meta-analysis. 1 Introduction NLP tools and online services such as the Stanford Parser or Google Translate are used for a wide va- riety of purposes and therefore also on very differ- ent kinds of data. Some use the Stanford Parser to parse literature (van Cranenburgh, 2012), while others use it for processing social media content (Brown, 2011). The parser, however, was not neces- sarily evaluated on literature or social media content during development. Still, users typically expect reasonable performance on any natural language in- put. This paper asks what we as developers can do to estimate the effect of a change to our system – not on the labeled test data that happens to be available to us, but on future, still unseen datasets provided by our end users. The usual practice in NLP is to evaluate a sys- tem on a small sample of held-out labeled data. The observed effect size on this sample is then val- idated by significance testing across data points, testing whether the observed difference in perfor- mance means is likely to be due to mere chance. The preferred significance test is probably the non- parametric paired bootstrap (Efron and Tibshirani, 1993; Berg-Kirkpatrick et al., 2012), but many re- searchers also resort to Student’s t-test for depen- dent means relying on the assumption that their met- ric scores are normally distributed. Such significance tests tell us nothing about how likely our change to our system is to lead to improve- ments on new datasets. The significance tests all rely on the assumption that our datapoints are sampled i.i.d. at random. The significance tests only tell us how likely it is that the observed difference in per- formance means would change if we sampled a big- ger test sample the same way we sampled the one we have available to us right now. In standard machine learning papers a similar sit- uation arises. …

Content truncated to 500 words