Scalable Decipherment for Machine Translation via Hash Sampling

Proceedings of the 51st Annual Meeting of the Association for Computational Linguistics, pages 362–371, Sofia, Bulgaria, August 4-9 2013. c⃝2013 Association for Computational Linguistics Scalable Decipherment for Machine Translation via Hash Sampling Sujith Ravi Google Mountain View, CA 94043 sravi@gooogle.com Abstract In this paper, we propose a new Bayesian inference method to train statistical ma- chine translation systems using only non- parallel corpora. Following a probabilis- tic decipherment approach, we first intro- duce a new framework for decipherment training that is flexible enough to incorpo- rate any number/type of features (besides simple bag-of-words) as side-information used for estimating translation models. In order to perform fast, efficient Bayesian inference in this framework, we then de- rive a hash sampling strategy that is in- spired by the work of Ahmed et al. (2012). The new translation hash sampler enables us to scale elegantly to complex mod- els (for the first time) and large vocab- ulary/corpora sizes. We show empirical results on the OPUS data—our method yields the best BLEU scores compared to existing approaches, while achieving sig- nificant computational speedups (several orders faster). We also report for the first time—BLEU score results for a large- scale MT task using only non-parallel data (EMEA corpus). 1 Introduction Statistical machine translation (SMT) systems these days are built using large amounts of bilin- gual parallel corpora. The parallel corpora are used to estimate translation model parameters in- volving word-to-word translation tables, fertilities, distortion, phrase translations, syntactic transfor- mations, etc. But obtaining parallel data is an ex- pensive process and not available for all language pairs or domains. On the other hand, monolin- gual data (in written form) exists and is easier to obtain for many languages. Learning translation models from monolingual corpora could help ad- dress the challenges faced by modern-day MT sys- tems, especially for low resource language pairs. Recently, this topic has been receiving increasing attention from researchers and new methods have been proposed to train statistical machine trans- lation models using only monolingual data in the source and target language. The underlying moti- vation behind most of these methods is that statis- tical properties for linguistic elements are shared across different languages and some of these sim- ilarities (mappings) could be automatically identi- fied from large amounts of monolingual data. The MT literature does cover some prior work on extracting or augmenting partial lexicons using non-parallel corpora (Rapp, 1995; Fung and McK- eown, 1997; Koehn and Knight, 2000; Haghighi et al., 2008). However, none of these meth- ods attempt to train end-to-end MT models, in- stead they focus on mining bilingual lexicons from monolingual corpora and often they require par- allel seed lexicons as a starting point. Some of them (Haghighi et al., 2008) also rely on addi- tional linguistic knowledge such as orthography, etc. to mine word translation pairs across related languages (e.g., Spanish/English). Unsupervised training methods have also been proposed in the past for related problems in decipherment (Knight and Yamada, 1999; Snyder et al., 2010; Ravi and Knight, 2011a) …

Content truncated to 500 words