An Evaluation to Detect and Correct Erroneous Characters Wrongly Substituted, Deleted and Inserted in Japanese and English Sentences Using Markov Models

AN EVALUATION TO DETECT AND CORRECT ERRONEOUS CHARACTERS WRONGLY SUBSTITUTED, DELETED AND INSERTED IN JAPANESE AND ENGLISH SEN~IENCES USING MARKOV MODELS Tetsuo ARAKI]" Satoru It(EIIARAiI Nobuyuld TSUI'~AIIARA] Yasunori KOMATSU]' "l Faculty of Engineering, l'~ukui University I ukui, 910 21APAN "i'~ NTT Communication Science l,al)oratories 1-2356 q'a.ke Yokosnka-Shi 2a8-03 Japan ABSTRACT In optical character recognition and coni.in- uous speech recognition of a natural language, it has been diflicult to detect error characters which are wrongly deleted and inserted. ]n <>r- der to judge three types of the errors, which are characters wrongly substituted, deleted or inserted in a Japanese "bunsetsu" and an l';n- glish word, and to correct these errors, this paper proposes new methods using rn-th or- der Markov chain model for Japanese "l~anji- kana" characters and Fmglish alphabets, as- suming that Markov l)robability of a correct chain of syllables or "kanji-kana" characters is greater than that of erroneous chains. From the results of the experiments, it is concluded that the methods is usefld for de- tecting as well as correcting these errors in Japanese "bunsetsu" and English words. Key words: Markov model, error detection, error correction, bunsetsu, substitution, dele- tion, insertion 1 Introduction In order to improve the man-machine in- terface with computers, the <tevelopment of input devices such as optical cha.racter tea<l- ets (OCR) or speech recognition devices are expected, llowew;r, it is not easy to input Japanese sentences J)y these devices, because. they are written by many kinds of charac- ters, especially thousands of "kanji" charac- ters. The sentences input through an OCR. or a speech recognition device usuMly contain erroneous character strings. The techniques of natural language process- ing are expected to find and correct these er- rors. tIowever, since current technologies of natural language analysis have been developed for correct sentences, they cannot directly be applied to these problems. Up to now, statis- tical approaches have been made to this prob- lem. Markov mo<lels are considered to be one of" machine learning models, sinfilar to neural networks a.nd fuzzy models. They have been applied to character chains of natural lan- g,,a~ges (e.g.,l);nglish)[l],[2], a.nd to phoneme cha.ins in continuous speech. reco~gnition 3 . [41 . [ 1~1. ¢1' 2nd-orde.r Markov model nt bunsets',l is known to be useful to correct errors in "kanji- kana." "/m nsetsu" [(;],to choose a correct sylla- ble chain from Japa.nese syllable "bunsetsu" candidates [7], and to re(!nce the ambigui- ties in translation processing of non-segmented "kana." sentences into "kanji-kana" sentences [8]. The erroneous characters can be classilied Ul,O three types, lhe hrst is w~ongly recog- nized chal;aclers instead of correct (haracters. The second and the third are wrongly inserted and deleted (skipped) characters respectively. Markov chain mode.Is above mentioned were restricted to tind and correct the first type of errors[5],[6]. No method has been proposed for correcting errors of the second and the. third types. 'Phe. rea.son might be considered to be I.he di[ticulties of finding the error location and distinguishing between deletion and insertion er I'ors. On the other hand, …

Content truncated to 500 words