Learning Continuous Phrase Representations for Translation Modeling

Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics, pages 699–709, Baltimore, Maryland, USA, June 23-25 2014. c⃝2014 Association for Computational Linguistics Learning Continuous Phrase Representations for Translation Modeling Jianfeng Gao Xiaodong He Wen-tau Yih Li Deng Microsoft Research One Microsoft Way Redmond, WA 98052, USA {jfgao,xiaohe,scottyih,deng}@microsoft.com Abstract This paper tackles the sparsity problem in estimating phrase translation probabilities by learning continuous phrase representa- tions, whose distributed nature enables the sharing of related phrases in their represen- tations. A pair of source and target phrases are projected into continuous-valued vec- tor representations in a low-dimensional latent space, where their translation score is computed by the distance between the pair in this new space. The projection is performed by a neural network whose weights are learned on parallel training data. Experimental evaluation has been performed on two WMT translation tasks. Our best result improves the performance of a state-of-the-art phrase-based statistical machine translation system trained on WMT 2012 French-English data by up to 1.3 BLEU points. 1 Introduction The phrase translation model, also known as the phrase table, is one of the core components of phrase-based statistical machine translation (SMT) systems. The most common method of construct- ing the phrase table takes a two-phase approach (Koehn et al. 2003). First, the bilingual phrase pairs are extracted heuristically from an automat- ically word-aligned training data. The second phase, which is the focus of this paper, is parame- ter estimation where each phrase pair is assigned with some scores that are estimated based on counting these phrases or their words using the same word-aligned training data. Phrase-based SMT systems have achieved state-of-the-art performance largely due to the fact that long phrases, rather than single words, are used as translation units so that useful context in- formation can be captured in selecting translations. However, longer phrases occur less often in train- ing data, leading to a severe data sparseness prob- lem in parameter estimation. There has been a plethora of research reported in the literature on improving parameter estimation for the phrase translation model (e.g., DeNero et al. 2006; Wuebker et al. 2010; He and Deng 2012; Gao and He 2013). This paper revisits the problem of scoring a phrase translation pair by developing a Continu- ous-space Phrase Translation Model (CPTM). The translation score of a phrase pair in this model is computed as follows. First, we represent each phrase as a bag-of-words vector, called word vec- tor henceforth. We then project the word vector, in either the source language or the target lan- guage, into a respective continuous feature vector in a common low-dimensional space that is lan- guage independent. The projection is performed by a multi-layer neural network. The projected feature vector forms the continuous representa- tion of a phrase. Finally, the translation score of a source-target phrase pair is computed by the dis- tance between their feature vectors. The main motivation behind the CPTM is to alleviate the data sparseness problem associated with the traditional counting-based methods by …

Content truncated to 500 words