An Evaluation of a Method to Detect and Correct Erroneous Characters in Japanese input through an OCR using Markov Models

An :Evaluation of a Method to :Detect and Correct Erroneous Characters in Japanese input through an OCR using Markov Models Tetsuo Araki Fukui University 3-9-1 Bunkyo, Fukui-shi,Japan Satoru Ikehara NTT Communication Science Laboratories 1-2356 Take, Yokosuka-shi, Japan Abstract The "Selective Error Correction Method" to judge these three types of errors, and correct them, using ra-th order Markov chain model for Japanese 'kanji-kana' char- acters, has been proposed and shown to be useful to detect and correct errors gener- ated randomly (Araki et al., 1994). In this paper, this method is applied to detect and correct erroneous characters in Japanese text input through an OCR.. The method is confirmed to be also elfective to detect and correct the errors introduced by the OCR. 1 Introduction In order to improve the computers' man-machine tares'faces, input devices such as Optical Character Readers(OCR.) or speech recognition devices have been developed. However, text input through an OCR or a speech recognition device usually contains erroneous character strings. The erroneous characters can be classified into three types. The first is characters that have been recognized incorrectly, that is taken to be characters other than the correct characters. The second and the third are extra characters wrongly inserted and deleted (skipped) characters. Markov chain modeLs have been used to find and correct the first type of errors. Recently, the Selective Error Correction Method to judge the three types of the errors and correct correct these errors, using m-th order Markov chain model for Japanese 'kanji-kana' characters, has been proposed (Arak iet al., 1994). In this paper, the Selective Error Correction Method is applied to detect and correct erroneous characters in Japanese text input through an OCR. Nobuyuki Tsukahara Yasunori Komatsu Fukui University Fukui University 3-9-1 Bunkyo, 3~9-1 Bunkyo, Fukui-shi, Japan Fukui-shi, Japan 2 Experimental Procedure and Conditions An experimental procedure using the Selective Error Correction Method for erroneous Japanese phrases input through OCR is described in Fig.1. Japanese "kanffi-kana. Bunsetsu Data OCR Error Detection and Correction procedure using Markov Models Fig.1. An Experimental Procedure Using Japanese Phrase Input Through OCR. Corrected Japanese "kanj i - kana" Bunsetsu Data Experimental Conditions are denoted as follows: (1) The number of phrases used for statistics: 70 issues of a daily Japanese newspaper containing 283,963 phrases. (2) The number of phrases input through the OCR: lOOO phrases (a) The average length of phrase (in 'kanji-kana' characters): 6 (b) The size of character fonts: 8 point (c) The input method to the OCR: Fax 198 3 Basic Definitions and the Selective Error Correction Method using 2nd-Order Markov Model A Japanese sentence can be separated into syntactic units called phrases ( usually called "bunsetsu" ). Japanese phrases in a text can be divided into two types: correct phrases, erroneous phrases. The set of correct Japanese phrases is represented by Fc. The set of erroneous phrases is denoted by FE, and it is further divided into three types: The first is er- roneous phrases which contain erroneous characters substituted wrongly in the phrase, and …

Content truncated to 500 words