A Neural Network Transliteration Model in Low Resource Settings

A Target Attention Model for Neural Machine Translation Hideya Mino † ‡ mino.h-gq@nhk.or.jp Andrew Finch † andyfinch@hotmail.com Eiichiro Sumita † eiichiro.sumita@nict.go.jp † National Institute of Information and Communications Technology, 3-5 Hikaridai, Seika-cho, Soraku-gun, Kyoto 619-0289, JAPAN ‡ Tokyo Institute of Technology, 2-12-1 Ookayama, Meguro-ku, Meguro-ku, 13, 152-8550, JAPAN Abstract Neural Machine Translation (NMT) with an attention mechanism has shown promising results by utilizing word alignments between the source and target sentences. Typically, training of NMT proceeds token-by-token on the target side, where each token is predicted using only a vector representing the current hidden-state, and the previous token. However, this strategy has serious shortcomings originating the lack of information about the partial target sequence hy- pothesis; specifically, this can lead to source tokens being translated multiple times or remain- ing untranslated. To alleviate this problem, we introduce a target-side attention mechanism to exploit the generated target sequence of tokens more effectively. We calculate a target-side con- text vector using a recurrent neural network and feed it to an attention mechanism so that the decoder can pay more or less attention to each token in the partially generated target sequence when predicting the next target token. Experiments on three different English-to-Japanese translation tasks show improvements of 0.6-1.5 BLEU points. 1 Introduction Recently, Neural Machine Translation (NMT) (Kalchbrenner and Blunsom, 2013; Sutskever et al., 2014) has been growing in popularity due to its capacity to model the translation process end-to-end within a single probabilistic model, and its potential for higher performance com- pared to existing phrase-based statistical machine translation (SMT) (Koehn, 2004). There are some unique features of NMT models which pose significant challenges for machine transla- tion. One is that NMT systems exploit Long Short-Term Memory (LSTM) units (Hochreiter and Schmidhuber, 1997) (or the similar Gated Recurrent Units (GRUs) (Cho et al., 2014)) which allow the systems to capture long-distance dependencies better than vanilla RNNs. Another is the attention mechanism, whereby the decoder can attend directly to localized information from the source sequence of tokens for generating the target sequence (Bahdanau et al., 2015; Luong et al., 2015). NMT systems are generally trained to maximize the likelihood of generating the target sequence of tokens given the source sequence. In practice, each target token is generated conditioned on the vector representing the current hidden-state of the model, and the previously generated target token. NMT, however, has a serious drawback in that some input tokens are unnecessarily trans- lated or mistakenly left untranslated (Tu et al., 2016). Our hypothesis is that this is mainly Proceedings of MT Summit XVI, vol.1: Research Track Nagoya, Sep. 18-22, 2017 | p. 15 because the hidden state of the LSTM decoder is not sufficiently representing all the infor- mation concerning the generated target sequence of tokens. Our work therefore endeavors to alleviate this drawback by explicitly handing a summary of the target sequence generated so far, at each step in the decoding process. Although an LSTM is able to provide the function of a long-term memory, the …

Content truncated to 500 words