A Deep Learning Approach to Machine Transliteration

Proceedings of the Fourth Workshop on Statistical Machine Translation , pages 233–241, Athens, Greece, 30 March – 31 March 2009. c⃝2009 Association for Computational Linguistics A Deep Learning Approach to Machine Transliteration Thomas Deselaers and Saˇsa Hasan and Oliver Bender and Hermann Ney Human Language Technology and Pattern Recognition Group – RWTH Aachen University <surname>@cs.rwth-aachen.de Abstract In this paper we present a novel translit- eration technique which is based on deep belief networks. Common approaches use finite state machines or other meth- ods similar to conventional machine trans- lation. Instead of using conventional NLP techniques, the approach presented here builds on deep belief networks, a tech- nique which was shown to work well for other machine learning problems. We show that deep belief networks have cer- tain properties which are very interesting for transliteration and possibly also for translation and that a combination with conventional techniques leads to an im- provement over both components on an Arabic-English transliteration task. 1 Introduction Transliteration, i.e. the transcription of words such as proper nouns from one language into another or, more commonly from one alphabet into another, is an important subtask of machine translation (MT) in order to obtain high quality output. We present a new technique for transliteration which is based on deep belief networks (DBNs), a well studied approach in machine learning. Transliteration can in principle be considered to be a small-scale translation problem and, thus, some ideas presented here can be transferred to the ma- chine translation domain as well. Transliteration has been in use in machine trans- lation systems, e.g. Russian-English, since the ex- istence of the field of machine translation. How- ever, to our knowledge it was first studied as a machine learning problem by Knight and Graehl (1998) using probabilistic finite-state transducers. Subsequently, the performance of this system was greatly improved by combining different spelling and phonetic models (Al-Onaizan and Knight, 2002). Huang et al. (2004) construct a proba- bilistic Chinese-English edit model as part of a larger alignment solution using a heuristic boot- strapped procedure. Freitag and Khadivi (2007) propose a technique which combines conventional MT methods with a single layer perceptron. In contrast to these methods which strongly build on top of well-established natural language processing (NLP) techniques, we propose an al- ternative model. Our new model is based on deep belief networks which have been shown to work well in other machine learning and pattern recog- nition areas (cf. Section 2). Since translation and transliteration are closely related and translitera- tion can be considered a translation problem on the character level, we discuss various methods from both domains which are related to the proposed approach in the following. Neural networks have been used in NLP in the past, e.g. for machine translation (Asunci´on Casta˜no et al., 1997) and constituent parsing (Titov and Henderson, 2007). However, it might not be straight-forward to obtain good results us- ing neural networks in this domain. In general, when training a neural network, one has to choose the structure …

Content truncated to 500 words