Does Multi-Encoder Help? A Case Study on Context-Aware Neural Machine Translation

Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 3512–3518 July 5 - 10, 2020. c⃝2020 Association for Computational Linguistics 3512 Does Multi-Encoder Help? A Case Study on Context-Aware Neural Machine Translation Bei Li1, Hui Liu1, Ziyang Wang1, Yufan Jiang1, Tong Xiao1,2∗, Jingbo Zhu1,2, Tongran Liu3, Changliang Li4 1NLP Lab, Northeastern University, Shenyang, China 2NiuTrans Research, Shenyang, China 3CAS Key Laboratory of Behavioral Science, Institute of Psychology, CAS, Beijing, China 4Kingsoft AI Lab, Beijing, China {libei neu,jiangyufan2018}@outlook.com, {huiliu,wangziyang}@stumail.neu.edu.cn, {xiaotong,zhujingbo}@mail.neu.edu.com, liutr@psych.ac.cn,lichangliang@kingsoft.com Abstract In encoder-decoder neural models, multiple encoders are in general used to represent the contextual information in addition to the indi- vidual sentence. In this paper, we investigate multi-encoder approaches in document-level neural machine translation (NMT). Surprising- ly, we find that the context encoder does not only encode the surrounding sentences but al- so behaves as a noise generator. This makes us rethink the real benefits of multi-encoder in context-aware translation - some of the im- provements come from robust training. We compare several methods that introduce noise and/or well-tuned dropout setup into the train- ing of these encoders. Experimental result- s show that noisy training plays an importan- t role in multi-encoder-based NMT, especial- ly when the training data is small. Also, we establish a new state-of-the-art on IWSLT Fr- En task by careful use of noise generation and dropout methods. 1 Introduction Sentence-level neural machine translation (NMT) systems ignore the discourse phenomena and en- code the individual source sentences with no use of contexts. In recent years, the context-aware models which learn contextual information from surround- ing sentences have shown promising results in gen- erating consistent and coherent translations (Zhang et al., 2018; Voita et al., 2018; Kim et al., 2019; Voita et al., 2019; Bawden et al., 2018; Miculicich et al., 2018; Maruf and Haffari, 2018; Maruf et al., 2019). There are two common approaches to incorpo- rating contexts into NMT: the simple way is to concatenate the context and the current sentence ∗Corresponding author. to form a context-aware input sequence (Agrawal et al., 2018; Tiedemann and Scherrer, 2017), where- as a more widely-used approach utilizes additional neural networks to encode context sentences (Jean et al., 2017; Voita et al., 2018; Zhang et al., 2018). Here we name the former as the single-encoder approach and name the latter as the multi-encoder approach. However, large-scale document corpora are not easily available. Most context-aware NMT systems are evaluated on small datasets and sig- nificant BLEU improvements are reported (Wang et al., 2017; Zhang et al., 2018; Tu et al., 2018). In our experiments, we find that the improvement per- sists if we feed pseudo sentences into the context encoder, especially when we train the system on small-scale data. A natural question here is: How much does the improvement come from the leverage of contextual information in multi-encoder? In this work, we aim to investigate what kind- s of information that the context-aware model captures. We re-implement several widely used context-aware architectures based on the multi- …

Content truncated to 500 words