A simple real-word error detection and correction using local word bigram and trigram

A simple real-word error detection and correction using local word bigram and trigram Pratip Samanta Computer Vision and Pattern Recognition Unit Indian Statistical Institute, Kolkata pratipsamanta@gmail.com Bidyut B. Chaudhuri Computer Vision and Pattern Recognition Unit Indian Statistical Institute, Kolkata b bcisical@gmail.com Abstract Spelling error is broadly classified in two categories namely non word error and real word error. In this paper a localized real word error detection and correction method is proposed where the scores of bigrams generated by immediate left and right neighbour of the candidate word and the trigram of these three words are combined. A single character position error model is assumed so that if a word W is erroneous then the correct word belongs to the set of real words S generated by single character edit operation on W. The above combined score is calculated also on all members of S. These words are ranked in the decreasing order of the score. By observing the rank and using a rule based approach, the error decision and correction candidates are simultaneously selected. The approach gives comparable accuracy with other existing approaches but is computationally attractive. Since only left and right neighbor are involved, multiple errors in a sentence can also be detected ( if the error occurs in every alternate words ). Keywords: Real word error, Local context. 1. Introduction Word error is a major hindrance to the real world applications of Natural Language Processing. In textual documents, word-error can be of two types. One is non-word error which has no meaning and other is real word error which is meaningful but not the intended word in the context of the sentence. Of these, non-word has been widely studied and algorithms to detect and suggest correction word for the error have been proposed. These algorithms are generally termed as spell-checker, which are integrated in various word- processing software like Microsoft Word1, LibreOffice Writer2, Ispell3, Aspell4 etc. For error occurring at two positions of a word, the commercial spell checkers work fairly well. Some studies on spell checking approaches are found in [1-5], that include English and non-English language like Bangla. However, the problem of real-word error is a more complex one. Usually, such error disturbs the syntax and semantics of the whole sentence, which requires human-being to detect it. However, an automatic syntactic/semantic analysis of a 'correct' sentence itself is a difficult 1 Microsoft Word is a word processor developed by Microsoft. 2 LibreOffice Writer a free open-source word processor. 3 Ispell is spell-checker for Unix. 4 Aspell is a free spell-checker for GNU software system. Proceedings of the Twenty-Fifth Conference on Computational Linguistics and Speech Processing (ROCLING 2013) 211 task and the analysis of an 'erroneous' sentence is almost impossible in most cases. Any word-error can be represented in terms of insertion, deletion or substitution of one or more character. If we consider 'space' as one character, the problem can become more complex. For example, the word 'within' can become 'with' and 'in' if a 'space' …

Content truncated to 500 words