Orthographic Disambiguation Incorporating Transliterated Probability

Orthographic Disambiguation Incorporating Transliterated Probability Eiji ARAMAKI Takeshi IMAI Kengo Miyo Kazuhiko Ohe University of Tokyo 7-3-1 Hongo, Bunkyo-ku, Tokyo 113-8655, Japan aramaki@hcc.h.u-tokyo.ac.jp Abstract Orthographic variance is a fundamental problem for many natural language process- ing applications. The Japanese language, in particular, contains many orthographic vari- ants for two main reasons: (1) transliterated words allow many possible spelling varia- tions, and (2) many characters in Japanese nouns can be omitted or substituted. Pre- vious studies have mainly focused on the former problem; in contrast, this study has addressed both problems using the same framework. First, we automatically col- lected both positive examples (sets of equiv- alent term pairs) and negative examples (sets of inequivalent term pairs). Then, by using both sets of examples, a support vector ma- chine based classifier determined whether two terms (t1 and t2) were equivalent. To boost accuracy, we added a transliterated probability P(t1|s)P(t2|s), which is the probability that both terms (t1 and t2) were transliterated from the same source term (s), to the machine learning features. Exper- imental results yielded high levels of ac- curacy, demonstrating the feasibility of the proposed approach. 1 Introduction Spelling variations, such as “center” and “centre”, which have different spellings but identical mean- ings, are problematic for many NLP applications including information extraction (IE), question an- swering (QA), and machine transliteration (MT). In Table 1: Examples of Orthographic Variants. spaghetti Thompson operation * ⟨⟩ indicates a pronunciation. () indicates a translation. this paper, these variations can be termed ortho- graphic variants. The Japanese language, in particular, contains many orthographic variants, for two main reasons: 1. It imports many words from other languages using transliteration, resulting in many possible spelling variations. For example, Masuyama et al. (2004) found at least six different spellings for“ spaghetti ” in newspaper articles (Table 1 Left). 2. Many characters in Japanese nouns can be omitted or substituted, leading to tons of in- sertion variations (Daille et al., 1996) (Table 1 Right). To address these problems, this study developed a support vector machine (SVM) based classifier that 48 can determine whether two terms are equivalent. Be- cause a SVM-based approach requires positive and negative examples, we also developed a method to automatically generate both examples. Our proposed method differs from previously de- veloped methods in two ways. 1. Previous studies have focused solely on the for- mer problem (transliteration); our target scope is wider. We addressed both transliteration and character omissions/substitutions using the same framework. 2. Most previous studies have focused on back- transliteration (Knight and Graehl, 1998; Goto et al., 2004), which has the goal of generating a source word (s) for a Japanese term (t). In con- trast, we employed a discriminative approach, which has the goal of determining whether two terms (t1 and t2) are equivalent. These two goals are related. For example, if two terms (t1 and t2) were transliterated from the same word (s), they should be orthographic variants. To incorporate this information, we incorporated a transliterated-probability (P(s|t1) × P(s|t2)) into …

Content truncated to 500 words