Novel Probabilistic Finite-State Transducers for Cognate and Transliteration Modeling

Novel Probabilistic Finite-State Transducers for Cognate and Transliteration Modeling Charles Schafer Department of Computer Science Center for Language and Speech Processing Johns Hopkins University Baltimore, MD 21218 USA cschafer@cs.jhu.edu Abstract We present and empirically compare a range of novel probabilistic finite-state transducer (PFST) models tar- geted at two major natural language string transduction tasks, transliteration selection and cognate translation se- lection. Evaluation is performed on 10 distinct language pair data sets, and in each case novel models consistently and substantially outperform a well-established standard reference algorithm. 1 Introduction This paper presents and empirically compares a range of novel probabilistic finite-state transducer (PFST) models targeted at two major natural lan- guage string transduction tasks. Four distinctly original transducer models are introduced: sym- bol networks, interlingua transducers, the joint dis- tribution/conditional operations tranducer, and ac- quired alphabetic identity transducers. These trans- duction models, some variations on previously pub- lished models, and also a standard reference algo- rithm, the probabilistic memoryless transducer in- troduced by Ristad and Yianilos (1997), are applied to two important problems in NLP: transliteration selection and cognate translation selection. Evaluation is performed on 10 distinct language pair data sets, and in each case novel models consistently and substantially outperform a well- established standard reference algorithm. This work is distinguished by the variety of tasks and languages addressed. Transliteration selec- tion is evaluated on 2 language pairs (Inuktitut- English and Arabic-English). Cognate selection is evaluated on a total of 8 distinct language pairs drawn from the Romance, Slavic, Turkic, Ger- manic, and North Indian language groups. The contribution of this work lies not in attempting to model in detail the complexities of a particular gen- erative process for a particular task. Rather, the goal was to investigate which general structures for transduction are effective over a reasonable vari- ety of natural language monotonic string transduc- tion tasks. The model we later denote the “Condi- tional Distribution/Unconditional Insertions” trans- ducer (CDUI) consistently figures among the most successful across tasks and language testbeds. The acquired alphabetic identity (AI) model also per- formed extremely well, achieving the highest per- formance of any model on 6 out of the total of 10 experiments across all tasks and language. Transliteration Selection: Examples Name from Source Language Rendering English Corpus Arabic Jensen JEEM NOON SEEN NOON Virginia FEH REH JEEM YEH NOON YEH ALEF William WAW LAM YEH ALEF MEEM Inuktitut Chretien kurittian Chartrand saaturaan Table 1: Instances of transliteration drawn from Arabic and Inuktitut. English Hindi Nepali Bengali Gujarati Punjabi large bilingual dictionary The Bridge Language Paradigm cognate similarity models or translation system Figure 1: Intra-language-family cognate selection as a part of the translation bridge to English. Transliteration selection is an important subtask in machine translation. This paper focuses on or- thographic transductions for transliteration model- ing (as opposed to phonemic ones). The transliter- ation task this paper uses for relative comparison of transduction models is that of selecting the correct English word corresponding to a back-transliterated 203 Proceedings of the 7th Conference of the …

Content truncated to 500 words