Why Nitpicking Works: Evidence for Occam’s Razor in Error Correctors

Why Nitpicking Works: Evidence for Occam’s Razor in Error Correctors Dekai WU†1 Grace NGAI‡2 Marine CARPUAT† dekai@cs.ust.hk csgngai@polyu.edu.hk marine@cs.ust.hk † Human Language Technology Center HKUST Department of Computer Science University of Science and Technology Clear Water Bay, Hong Kong ‡ Hong Kong Polytechnic University Department of Computing Kowloon Hong Kong Abstract Empirical experience and observations have shown us when powerful and highly tunable classifiers such as maximum en- tropy classifiers, boosting and SVMs are applied to language processing tasks, it is possible to achieve high accuracies, but eventually their performances all tend to plateau out at around the same point. To further improve performance, various error correction mechanisms have been developed, but in practice, most of them cannot be relied on to predictably improve per- formance on unseen data; indeed, depending upon the test set, they are as likely to degrade accuracy as to improve it. This problem is especially severe if the base classifier has already been finely tuned. In recent work, we introduced N-fold Templated Piped Cor- rection, or NTPC (“nitpick”), an intriguing error corrector that is designed to work in these extreme operating conditions. De- spite its simplicity, it consistently and robustly improves the ac- curacy of existing highly accurate base models. This paper in- vestigates some of the more surprising claims made by NTPC, and presents experiments supporting an Occam’s Razor argu- ment that more complex models are damaging or unnecessary in practice. 1 Introduction The investigation we describe here arose from a very commonly discussed experience, apparently triggered by the recent popularity of shared task evaluations that have opened opportunities for researchers to informally com- pare their experiences “with a common denominator”, so to speak. Among the perennial observations which are made during the analysis of the results is that (1) methods de- signed to “fine-tune” the high-accuracy base classifiers behave unpredictably, their success or failure often ap- pearing far more sensitive to where the test set was drawn from, rather than on any true quality of the “fine-tuning”, and consequently, (2) the resulting system rankings are often unpredictable, especially as they are typically con- ducted only on a single new test set, often drawn from a single arbitrary new source of a significantly different nature than the training sets. One could argue that such evaluations do not constitute a fair test, but in fact, this is 1The author would like to thank the Hong Kong Research Grants Council (RGC) for supporting this research in part through research grants RGC6083/99E, RGC6256/00E, and DAG03/04.EG09. 2The author would like to thank the Hong Kong Polytechnic Univer- sity for supporting this research in part through research grants A-PE37 and 4-Z03S. where computational linguistics modeling diverges from machine learning theory, since for any serious NLP ap- plication, such evaluations constitute a much more accu- rate representation of the real world. We believe one primary reason for this common ex- perience is that the models involved are typically al- ready operating well beyond the limits of accuracy of the models’ …

Content truncated to 500 words