Sentence-Level Agreement for Neural Machine Translation

Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 3076–3082 Florence, Italy, July 28 - August 2, 2019. c⃝2019 Association for Computational Linguistics 3076 Sentence-Level Agreement for Neural Machine Translation Mingming Yang1∗, Rui Wang2, Kehai Chen2, Masao Utiyama2, Eiichiro Sumita2, Min Zhang1,3, and Tiejun Zhao1 1School of Computer Science and Technology, Harbin Institute of Technology, Harbin, China 2National Institute of Information and Communications Technology (NICT), Kyoto, Japan 3School of Computer Science and Technology, Soochow University, Suzhou, China mmyang@hit-mtlab.net, minzhang@suda.edu.cn, tjzhao@hit.edu.cn {wangrui, khchen, mutiyama, eiichiro.sumita}@nict.go.jp Abstract The training objective of neural machine translation (NMT) is to minimize the loss between the words in the translated sentences and those in the references. In NMT, there is a natural correspondence between the source sentence and the target sentence. However, this relationship has only been represented using the entire neural network and the training objective is computed in word- level. In this paper, we propose a sentence- level agreement module to directly minimize the difference between the representation of source and target sentence. The proposed agreement module can be integrated into NMT as an additional training objective function and can also be used to enhance the representation of the source sentences. Empirical results on the NIST Chinese-to-English and WMT English-to-German tasks show the proposed agreement module can significantly improve the NMT performance. 1 Introduction Neural network based methods have been applied to several natural language processing tasks (Zhang et al., 2016; Li et al., 2018; Chen et al., 2018; Li et al., 2019; He et al., 2018). In neural machine translation (NMT), unlike conventional phrase-based statistical machine translation, an attention mechanism is adopted to help align output with input words (Bahdanau et al., 2015). It is based on the estimation of a probability distribution over all input words for each target word. However, source and target words are in different representation space, and they still have to go through a long information processing procedure that may lead to the source words are incorrectly translated into the target words. ∗Mingming Yang was an internship research fellow at NICT when conducting this work. Based on this hypothesis, Kuang et al. (2018) proposed a direct bridging model, which directly connects source and target word embeddings seeking to minimize errors in the translation. Tu et al. (2017) incorporated a reconstructor module into NMT, which reconstructs the input source sentence from the hidden layer of the output target sentence to enhance source representation. However, in previous studies, the training objective function was usually based on word-level and lacked explicit sentence- level relationships (Zhang and Zhao, 2019). Although Transformer model (Vaswani et al., 2017) has archived state-of-the-art performance of NMT, more attention is paid to the words-level relationship via self-attention networks. Sentence-level agreement method has been applied to many natural language processing tasks. Aliguliyev (2009) used sentence similarity measure technique for automatic text summarization. Liang et al. (2010) have shown that the sentence similarity algorithm based on VSM is beneficial to address the FAQ problem. Su …

Content truncated to 500 words