Creating Robust Supervised Classifiers via Web-Scale N-Gram Data

Proceedings of the 48th Annual Meeting of the Association for Computational Linguistics, pages 865–874, Uppsala, Sweden, 11-16 July 2010. c⃝2010 Association for Computational Linguistics Creating Robust Supervised Classifiers via Web-Scale N-gram Data Shane Bergsma University of Alberta sbergsma@ualberta.ca Emily Pitler University of Pennsylvania epitler@seas.upenn.edu Dekang Lin Google, Inc. lindek@google.com Abstract In this paper, we systematically assess the value of using web-scale N-gram data in state-of-the-art supervised NLP classifiers. We compare classifiers that include or ex- clude features for the counts of various N-grams, where the counts are obtained from a web-scale auxiliary corpus. We show that including N-gram count features can advance the state-of-the-art accuracy on standard data sets for adjective order- ing, spelling correction, noun compound bracketing, and verb part-of-speech dis- ambiguation. More importantly, when op- erating on new domains, or when labeled training data is not plentiful, we show that using web-scale N-gram features is essen- tial for achieving robust performance. 1 Introduction Many NLP systems use web-scale N-gram counts (Keller and Lapata, 2003; Nakov and Hearst, 2005; Brants et al., 2007). Lapata and Keller (2005) demonstrate good performance on eight tasks using unsupervised web-based models. They show web counts are superior to counts from a large corpus. Bergsma et al. (2009) propose un- supervised and supervised systems that use counts from Google’s N-gram corpus (Brants and Franz, 2006). Web-based models perform particularly well on generation tasks, where systems choose between competing sequences of output text (such as different spellings), as opposed to analysis tasks, where systems choose between abstract la- bels (such as part-of-speech tags or parse trees). In this work, we address two natural and related questions which these previous studies leave open: 1. Is there a benefit in combining web-scale counts with the features used in state-of-the- art supervised approaches? 2. How well do web-based models perform on new domains or when labeled data is scarce? We address these questions on two generation and two analysis tasks, using both existing N-gram data and a novel web-scale N-gram corpus that includes part-of-speech information (Section 2). While previous work has combined web-scale fea- tures with other features in specific classification problems (Modjeska et al., 2003; Yang et al., 2005; Vadas and Curran, 2007b), we provide a multi-task, multi-domain comparison. Some may question why supervised approaches are needed at all for generation problems. Why not solely rely on direct evidence from a giant cor- pus? For example, for the task of prenominal ad- jective ordering (Section 3), a system that needs to describe a ball that is both big and red can sim- ply check that big red is more common on the web than red big, and order the adjectives accordingly. It is, however, suboptimal to only use N-gram data. For example, ordering adjectives by direct web evidence performs 7% worse than our best supervised system (Section 3.2). No matter how large the web becomes, there will always be plau- sible constructions that never occur. For example, there are currently no pages indexed by Google with the …

Content truncated to 500 words