Real-Word Spelling Correction using Google Web 1T 3-grams

Proceedings of the 2009 Conference on Empirical Methods in Natural Language Processing, pages 1241–1249, Singapore, 6-7 August 2009. c⃝2009 ACL and AFNLP Real-Word Spelling Correction using Google Web 1T 3-grams Aminul Islam Department of Computer Science University of Ottawa Ottawa, ON, K1N 6N5, Canada mdislam@site.uottawa.ca Diana Inkpen Department of Computer Science University of Ottawa Ottawa, ON, K1N 6N5, Canada diana@site.uottawa.ca Abstract We present a method for detecting and correcting multiple real-word spelling er- rors using the Google Web 1T 3-gram data set and a normalized and modified ver- sion of the Longest Common Subsequence (LCS) string matching algorithm. Our method is focused mainly on how to im- prove the detection recall (the fraction of errors correctly detected) and the correc- tion recall (the fraction of errors correctly amended), while keeping the respective precisions (the fraction of detections or amendments that are correct) as high as possible. Evaluation results on a standard data set show that our method outperforms two other methods on the same task. 1 Introduction Real-word spelling errors are words in a text that occur when a user mistakenly types a correctly spelled word when another was intended. Errors of this type may be caused by the writer’s igno- rance of the correct spelling of the intended word or by typing mistakes. Such errors generally go unnoticed by most spellcheckers as they deal with words in isolation, accepting them as correct if they are found in the dictionary, and flagging them as errors if they are not. This approach would be sufficient to detect the non-word error myss in “It doesn’t know what the myss is all about.” but not the real-word error muss in “It doesn’t know what the muss is all about.” To detect the latter, the spell-checker needs to make use of the sur- rounding context such as, in this case, to recog- nise that fuss is more likely to occur than muss in the context of all about. Ironically, errors of this type may even be caused by spelling checkers in the correction of non-word spelling errors when the auto-correct feature in some word-processing software sometimes silently change a non-word to the wrong real word (Hirst and Budanitsky, 2005), and sometimes when correcting a flagged error, the user accidentally make a wrong selection from the choices offered (Wilcox-O’Hearn et al., 2008). An extensive review of real-word spelling cor- rection is given in (Pedler, 2007; Hirst and Budan- itsky, 2005) and the problem of spelling correction more generally is reviewed in (Kukich, 1992). The Google Web 1T data set (Brants and Franz, 2006), contributed by Google Inc., contains En- glish word n-grams (from unigrams to 5-grams) and their observed frequency counts calculated over 1 trillion words from web page text col- lected by Google in January 2006. The text was tokenised following the Penn Treebank tokenisa- tion, except that hyphenated words, dates, email addresses and URLs are kept as single tokens. The sentence boundaries are marked with two spe- cial tokens <S> and </S>. Words …

Content truncated to 500 words