The Web as a Baseline: Evaluating the Performance of Unsupervised Web-based Models for a Range of NLP Tasks

The Web as a Baseline: Evaluating the Performance of Unsupervised Web-based Models for a Range of NLP Tasks Mirella Lapata Department of Computer Science University of Sheffield 211 Portobello St., Sheffield S1 4DP mlap@dcs.shef.ac.uk Frank Keller School of Informatics University of Edinburgh 2 Buccleuch Pl., Edinburgh EH8 9LW keller@inf.ed.ac.uk Abstract Previous work demonstrated that web counts can be used to approximate bigram frequen- cies, and thus should be useful for a wide va- riety of NLP tasks. So far, only two gener- ation tasks (candidate selection for machine translation and confusion-set disambiguation) have been tested using web-scale data sets. The present paper investigates if these results gener- alize to tasks covering both syntax and seman- tics, both generation and analysis, and a larger range of n-grams. For the majority of tasks, we fi nd that simple, unsupervised models perform better when n-gram frequencies are obtained from the web rather than from a large corpus. However, in most cases, web-based models fail to outperform more sophisticated state-of-the- art models trained on small corpora. We ar- gue that web-based models should therefore be used as a baseline for, rather than an alternative to, standard models. 1 Introduction Keller and Lapata (2003) investigated the validity of web counts for a range of predicate-argument bigrams (verb- object, adjective-noun, and noun-noun bigrams). They presented a simple method for retrieving bigram counts from the web by querying a search engine and demon- strated that web counts (a) correlate with frequencies ob- tained from a carefully edited, balanced corpus such as the 100M words British National Corpus (BNC), (b) cor- relate with frequencies recreated using smoothing meth- ods in the case of unseen bigrams, (c) reliably predict hu- man plausibility judgments, and (d) yield state-of-the-art performance on pseudo-disambiguation tasks. Keller and Lapata’s (2003) results suggest that web- based frequencies can be a viable alternative to bigram frequencies obtained from smaller corpora or recreated using smoothing. However, they do not demonstrate that realistic NLP tasks can benefi t from web counts. In or- der to show this, web counts would have to be applied to a diverse range of NLP tasks, both syntactic and seman- Task n POS Ling Type MT candidate select. 1,2 V, N Sem Generation Spelling correction 1,2,3 Any Syn/Sem Generation Adjective ordering 1,2 Adj Sem Generation Compound bracketing 1,2 N Syn Analysis Compound interpret. 1,2,3 N, P Sem Analysis Countability detection 1,2 N, Det Sem Analysis Table 1: Overview of the tasks investigated in this paper (n: size of n-gram; POS: parts of speech; Ling: linguistic knowledge; Type: type of task) tic, involving analysis (e.g., disambiguation) and gener- ation (e.g., selection among competing outputs). Also, it remains to be shown that the web-based approach scales up to larger n-grams (e.g., trigrams), and to combinations of different parts of speech (Keller and Lapata 2003 only tested bigrams involving nouns, verbs, and adjectives). Another important question is whether web-based meth- ods, which are by defi nition unsupervised, can be com- petitive alternatives to supervised approaches used …

Content truncated to 500 words