Learning bilingual phrase representations with recurrent neural networks

Learning Bilingual Phrase Representations with Recurrent Neural Networks Hideya Mino hideya.mino@nict.go.jp Andrew Finch andrew.finch@nict.go.jp Eiichiro Sumita eiichiro.sumita@nict.go.jp National Institute of Information and Communications Technology, 3-5 Hikaridai, Seika-cho, Soraku-gun, Kyoto 619-0289, JAPAN Abstract We introduce a novel method for bilingual phrase representation with Recurrent Neural Net- works (RNNs), which transforms a sequence of word feature vectors into a fixed-length phrase vector across two languages. Our method measures the difference between the vectors of source- and target-side phrases, and can be used to predict the semantic equivalence of source and target word sequences in the phrasal translation units used in phrase-based statistical ma- chine translation. Our experiments show that the proposed method is effective in a bilingual phrasal semantic equivalence determination task and a machine translation task. 1 Introduction In recent years, continuous vector representations of word, phrase, and sentence which allevi- ate issues of sparsity have successfully been used in a number of natural language processing tasks. Language models with continuous word representations (Bengio et al., 2003; Mikolov et al., 2010; Mikolov, 2012) based on neural networks have outperformed the previous state- of-the-art approaches. These language models map each word to a dense, low-dimensional, real-valued vector, and estimate the probability of words in a continuous space. Representa- tions for phrases have been used in the context of Statistical Machine Translation (SMT). Zou et al. (2013) used phrasal representations for computing the distance between phrase pairs and added a feature based on this distance into the log-linear model of a phrase-based SMT system (Koehn et al., 2003). Their method learned bilingual word representations, and subsequently obtained the phrase-level representations by simply averaging word vectors. Continuous rep- resentations for phrases or sentences with neural networks – such as a feed-forward, recursive, or recurrent neural networks – have also been used in SMT. A phrase representation model using a feed-forward neural network for phrase-based SMT was proposed by Schwenk (2007, 2012), and achieved significant BLEU score improvements. Since the model directly projects feature vectors not from words but from phrases or sentences onto a continuous vector space, the representations can contain more global semantic information. In this paper, we propose a new method to learn bilingual phrase representations for phrase- based SMT using two Recurrent Neural Networks (RNNs) (source- and target-side) combined in a simple linear architecture. We follow the idea of Cho et al. (2014) that the last hidden state of the RNN is a summary representation of the whole input phrase, and the summary representations with the same meaning are trained to be the same vector representation. The procedure is similar to that used in the RNN of Cho et al. (2014), which predicts the next word Proceedings of MT Summit XV, vol.1: MT Researchers' Track Miami, Oct 30 - Nov 3, 2015 | p. 44 in the sequence with a conditional probability. In contrast to this, our model uses an objective with a similarity distance instead of a conditional probability, and learns to minimize the error distance. Furthermore, we developed a …

Content truncated to 500 words