Unifying Bayesian Inference and Vector Space Models for Improved Decipherment

Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing, pages 836–845, Beijing, China, July 26-31, 2015. c⃝2015 Association for Computational Linguistics Unifying Bayesian Inference and Vector Space Models for Improved Decipherment Qing Dou∗, Ashish Vaswani∗, Kevin Knight Information Sciences Institute Department of Computer Science University of Southern California {qdou,avaswani,knight}@isi.edu Chris Dyer School of Computer Science Carnegie Mellon University cdyer@cs.cmu.edu Abstract We introduce into Bayesian decipherment a base distribution derived from similari- ties of word embeddings. We use Dirich- let multinomial regression (Mimno and McCallum, 2012) to learn a mapping be- tween ciphertext and plaintext word em- beddings from non-parallel data. Exper- imental results show that the base dis- tribution is highly beneficial to decipher- ment, improving state-of-the-art decipher- ment accuracy from 45.8% to 67.4% for Spanish/English, and from 5.1% to 11.2% for Malagasy/English. 1 Introduction Tremendous advances in Machine Translation (MT) have been made since we began applying automatic learning techniques to learn translation rules automatically from parallel data. However, reliance on parallel data also limits the develop- ment and application of high-quality MT systems, as the amount of parallel data is far from adequate in low-density languages and domains. In general, it is easier to obtain non-parallel monolingual data. The ability to learn transla- tions from monolingual data can alleviate obsta- cles caused by insufficient parallel data. Motivated by this idea, researchers have proposed different approaches to tackle this problem. They can be largely divided into two groups. The first group is based on the idea proposed by Rapp (1995), in which words are represented as context vectors, and two words are likely to be translations if their context vectors are simi- lar. Initially, the vectors contained only context ∗ Equal contribution words. Later extensions introduced more fea- tures (Haghighi et al., 2008; Garera et al., 2009; Bergsma and Van Durme, 2011; Daum´e and Jagar- lamudi, 2011; Irvine and Callison-Burch, 2013b; Irvine and Callison-Burch, 2013a), and used more abstract representation such as word embeddings (Klementiev et al., 2012). Another promising approach to solve this prob- lem is decipherment. It has drawn significant amounts of interest in the past few years (Ravi and Knight, 2011; Nuhn et al., 2012; Dou and Knight, 2013; Ravi, 2013) and has been shown to improve end-to-end translation. Decipherment views a for- eign language as a cipher for English and finds a translation table that converts foreign texts into sensible English. Both approaches have been shown to improve quality of MT systems for domain adaptation (Daum´e and Jagarlamudi, 2011; Dou and Knight, 2012; Irvine et al., 2013) and low density lan- guages (Irvine and Callison-Burch, 2013a; Dou et al., 2014). Meanwhile, they have their own ad- vantages and disadvantages. While context vec- tors can take larger context into account, it re- quires high quality seed lexicons to learn a map- ping between two vector spaces. In contrast, de- cipherment does not depend on any seed lexicon, but only looks at a limited …

Content truncated to 500 words