Recurrent Neural Networks for Word Alignment Model

Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics, pages 1470–1480, Baltimore, Maryland, USA, June 23-25 2014. c⃝2014 Association for Computational Linguistics Recurrent Neural Networks for Word Alignment Model Akihiro Tamura∗, Taro Watanabe, Eiichiro Sumita National Institute of Information and Communications Technology 3-5 Hikaridai, Seika-cho, Soraku-gun, Kyoto, JAPAN a-tamura@ah.jp.nec.com, {taro.watanabe, eiichiro.sumita}@nict.go.jp Abstract This study proposes a word alignment model based on a recurrent neural net- work (RNN), in which an unlimited alignment history is represented by re- currently connected hidden layers. We perform unsupervised learning using noise-contrastive estimation (Gutmann and Hyv¨arinen, 2010; Mnih and Teh, 2012), which utilizes artificially generated negative samples. Our alignment model is directional, similar to the generative IBM models (Brown et al., 1993). To overcome this limitation, we encourage agreement between the two directional models by introducing a penalty function that en- sures word embedding consistency across two directional models during training. The RNN-based model outperforms the feed-forward neural network-based model (Yang et al., 2013) as well as the IBM Model 4 under Japanese-English and French-English word alignment tasks, and achieves comparable transla- tion performance to those baselines for Japanese-English and Chinese-English translation tasks. 1 Introduction Automatic word alignment is an important task for statistical machine translation. The most classical approaches are the probabilistic IBM models 1-5 (Brown et al., 1993) and the HMM model (Vogel et al., 1996). Various studies have extended those models. Yang et al. (2013) adapted the Context- Dependent Deep Neural Network for HMM (CD- DNN-HMM) (Dahl et al., 2012), a type of feed- forward neural network (FFNN)-based model, to ∗ The first author is now affiliated with Knowledge Discovery Research Laboratories, NEC Corporation, Nara, Japan. the HMM alignment model and achieved state-of- the-art performance. However, the FFNN-based model assumes a first-order Markov dependence for alignments. Recurrent neural network (RNN)-based models have recently demonstrated state-of-the-art per- formance that outperformed FFNN-based models for various tasks (Mikolov et al., 2010; Mikolov and Zweig, 2012; Auli et al., 2013; Kalchbrenner and Blunsom, 2013; Sundermeyer et al., 2013). An RNN has a hidden layer with recurrent con- nections that propagates its own previous signals. Through the recurrent architecture, RNN-based models have the inherent property of modeling long-span dependencies, e.g., long contexts, in in- put data. We assume that this property would fit with a word alignment task, and we propose an RNN-based word alignment model. Our model can maintain and arbitrarily integrate an alignment history, e.g., bilingual context, which is longer than the FFNN-based model. The NN-based alignment models are super- vised models. Unfortunately, it is usually dif- ficult to prepare word-by-word aligned bilingual data. Yang et al. (2013) trained their model from word alignments produced by traditional unsuper- vised probabilistic models. However, with this approach, errors induced by probabilistic mod- els are learned as correct alignments; thus, gen- eralization capabilities are limited. To solve this problem, we apply noise-contrastive estimation (NCE) (Gutmann and Hyv¨arinen, 2010; Mnih and Teh, 2012) for unsupervised training of our RNN-based model without gold standard align- ments or …

Content truncated to 500 words