Layer-Wise Multi-View Learning for Neural Machine Translation

Proceedings of the 28th International Conference on Computational Linguistics, pages 4275–4286 Barcelona, Spain (Online), December 8-13, 2020 4275 Layer-Wise Multi-View Learning for Neural Machine Translation Qiang Wang1∗, Changliang Li2, Yue Zhang3†, Tong Xiao4,5, Jingbo Zhu4,5 1Machine Intelligence Technology Lab, Alibaba DAMO Academy, Hangzhou, China 2Kingsoft AI Lab, Beijing, China 3Institute of Advanced Technology, Westlake Institute for Advanced Study, Hangzhou, China 4 School of Computer Science and Engineering, Northeastern University, Shenyang, China 5NiuTrans Research, Shenyang, China zhiniao.wq@alibaba-inc.com, lichangliang@kingsoft.com zhangyue@westlake.edu.cn, {xiaotong,zhujingbo}@mail.neu.edu.cn Abstract Traditional neural machine translation is limited to the topmost encoder layer’s context represen- tation and cannot directly perceive the lower encoder layers. Existing solutions usually rely on the adjustment of network architecture, making the calculation more complicated or introduc- ing additional structural restrictions. In this work, we propose layer-wise multi-view learning to solve this problem, circumventing the necessity to change the model structure. We regard each encoder layer’s off-the-shelf output, a by-product in layer-by-layer encoding, as the redundant view for the input sentence. In this way, in addition to the topmost encoder layer (referred to as the primary view), we also incorporate an intermediate encoder layer as the auxiliary view. We feed the two views to a partially shared decoder to maintain independent prediction. Consistency regularization based on KL divergence is used to encourage the two views to learn from each other. Extensive experimental results on five translation tasks show that our approach yields sta- ble improvements over multiple strong baselines. As another bonus, our method is agnostic to network architectures and can maintain the same inference speed as the original model. 1 Introduction Neural Machine Translation (NMT) adopts the encoder-decoder paradigm to model the entire translation process (Bahdanau et al., 2015). Specifically, the encoder finds a multi-layer representation of the source sentence, and the decoder queries the topmost encoding representation to produce the target sentence through a cross-attention mechanism (Wu et al., 2016; Vaswani et al., 2017). However, such over- reliance on the topmost encoding layer is problematic in two aspects: (1) Prone to over-fitting, especially when the encoder is under-trained, such as in low-resource tasks (Wang et al., 2018); (2) It cannot make full use of representations extracted from lower encoder layers, which are syntactically and semantically complementary to higher layers (Peters et al., 2018; Raganato and Tiedemann, 2018). Researchers have proposed many methods to make the model aware of various encoder layers besides the topmost to mitigate this issue. Almost all of them resort to the adjustment of network structure, which can be further divided into two categories. The first is to merge the feature representations extracted by distinct encoder layers before being fed to the decoder (Wang et al., 2018; Dou et al., 2018; Wang et al., 2019b). The differences between them lie in the design of the merge function: through self-attention (Wang et al., 2018), recurrent neural network (Wang et al., 2019b), or tree-like hierarchical merge (Dou et al., 2018). Moreover, the second makes each decoder layer explicitly align to a parallel encoder layer (He …

Content truncated to 500 words