Bilingually-constrained Phrase Embeddings for Machine Translation

Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics, pages 111–121, Baltimore, Maryland, USA, June 23-25 2014. c⃝2014 Association for Computational Linguistics Bilingually-constrained Phrase Embeddings for Machine Translation Jiajun Zhang1, Shujie Liu2, Mu Li2, Ming Zhou2 and Chengqing Zong1 1National Laboratory of Pattern Recognition, CASIA, Beijing, P.R. China {jjzhang,cqzong}@nlpr.ia.ac.cn 2Microsoft Research Asia, Beijing, P.R. China {shujliu,muli,mingzhou}@microsoft.com Abstract We propose Bilingually-constrained Re- cursive Auto-encoders (BRAE) to learn semantic phrase embeddings (compact vector representations for phrases), which can distinguish the phrases with differ- ent semantic meanings. The BRAE is trained in a way that minimizes the seman- tic distance of translation equivalents and maximizes the semantic distance of non- translation pairs simultaneously. After training, the model learns how to embed each phrase semantically in two languages and also learns how to transform semantic embedding space in one language to the other. We evaluate our proposed method on two end-to-end SMT tasks (phrase ta- ble pruning and decoding with phrasal se- mantic similarities) which need to mea- sure semantic similarity between a source phrase and its translation candidates. Ex- tensive experiments show that the BRAE is remarkably effective in these two tasks. 1 Introduction Due to the powerful capacity of feature learn- ing and representation, Deep (multi-layer) Neural Networks (DNN) have achieved a great success in speech and image processing (Kavukcuoglu et al., 2010; Krizhevsky et al., 2012; Dahl et al., 2012). Recently, statistical machine translation (SMT) community has seen a strong interest in adapting and applying DNN to many tasks, such as word alignment (Yang et al., 2013), translation confi- dence estimation (Mikolov et al., 2010; Liu et al., 2013; Zou et al., 2013), phrase reordering predic- tion (Li et al., 2013), translation modelling (Auli et al., 2013; Kalchbrenner and Blunsom, 2013) and language modelling (Duh et al., 2013; Vaswani et al., 2013). Most of these works attempt to im- prove some components in SMT based on word embedding, which converts a word into a dense, low dimensional, real-valued vector representation (Bengio et al., 2003; Bengio et al., 2006; Collobert and Weston, 2008; Mikolov et al., 2013). However, in the conventional (phrase-based) SMT, phrases are the basic translation units. The models using word embeddings as the direct in- puts to DNN cannot make full use of the whole syntactic and semantic information of the phrasal translation rules. Therefore, in order to success- fully apply DNN to model the whole translation process, such as modelling the decoding process, learning compact vector representations for the ba- sic phrasal translation units is the essential and fundamental work. In this paper, we explore the phrase embedding, which represents a phrase (sequence of words) with a real-valued vector. In some previous works, phrase embedding has been discussed from differ- ent views. Socher et al. (2011) make the phrase embeddings capture the sentiment information. Socher et al. (2013a) enable the phrase embed- dings to mainly capture the syntactic knowledge. Li et al. (2013) attempt to encode the reordering pattern in the phrase embeddings. Kalchbrenner and …

Content truncated to 500 words