Using the Web for Language Independent Spellchecking and Autocorrection

Proceedings of the 2009 Conference on Empirical Methods in Natural Language Processing, pages 890–899, Singapore, 6-7 August 2009. c⃝2009 ACL and AFNLP Using the Web for Language Independent Spellchecking and Autocorrection Casey Whitelaw and Ben Hutchinson and Grace Y Chung and Gerard Ellis Google Inc. Level 5, 48 Pirrama Rd, Pyrmont NSW 2009, Australia {whitelaw,benhutch,gracec,ged}@google.com Abstract We have designed, implemented and eval- uated an end-to-end system spellcheck- ing and autocorrection system that does not require any manually annotated train- ing data. The World Wide Web is used as a large noisy corpus from which we infer knowledge about misspellings and word usage. This is used to build an er- ror model and an n-gram language model. A small secondary set of news texts with artificially inserted misspellings are used to tune confidence classifiers. Because no manual annotation is required, our sys- tem can easily be instantiated for new lan- guages. When evaluated on human typed data with real misspellings in English and German, our web-based systems outper- form baselines which use candidate cor- rections based on hand-curated dictionar- ies. Our system achieves 3.8% total error rate in English. We show similar improve- ments in preliminary results on artificial data for Russian and Arabic. 1 Introduction Spellchecking is the task of predicting which words in a document are misspelled. These pre- dictions might be presented to a user by under- lining the misspelled words. Correction is the task of substituting the well-spelled hypotheses for misspellings. Spellchecking and autocorrec- tion are widely applicable for tasks such as word- processing and postprocessing Optical Character Recognition. We have designed, implemented and evaluated an end-to-end system that performs spellchecking and autocorrection. The key novelty of our work is that the sys- tem was developed entirely without the use of manually annotated resources or any explicitly compiled dictionaries of well-spelled words. Our multi-stage system integrates knowledge from sta- tistical error models and language models (LMs) with a statistical machine learning classifier. At each stage, data are required for training models and determining weights on the classifiers. The models and classifiers are all automatically trained from frequency counts derived from the Web and from news data. System performance has been validated on a set of human typed data. We have also shown that the system can be rapidly ported across languages with very little manual effort. Most spelling systems today require some hand- crafted language-specific resources, such as lex- ica, lists of misspellings, or rule bases. Sys- tems using statistical models require large anno- tated corpora of spelling errors for training. Our statistical models require no annotated data. In- stead, we rely on the Web as a large noisy corpus in the following ways. 1) We infer information about misspellings from term usage observed on the Web, and use this to build an error model. 2) The most frequently observed terms are taken as a noisy list of potential candidate corrections. 3) Token n-grams are used to build an LM, which we use to make context-appropriate corrections. …

Content truncated to 500 words