Learning to Faithfully Rationalize by Construction

Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 414–426 July 5 - 10, 2020. c⃝2020 Association for Computational Linguistics 414 Multiscale Collaborative Deep Models for Neural Machine Translation Xiangpeng Wei1,2∗, Heng Yu3†, Yue Hu1,2†, Yue Zhang4,5, Rongxiang Weng3, Weihua Luo3 1Institute of Information Engineering, Chinese Academy of Sciences, Beijing, China 2School of Cyber Security, University of Chinese Academy of Sciences, Beijing, China 3Machine Intelligence Technology Lab, Alibaba Group, Hangzhou, China 4School of Engineering, Westlake University, Hangzhou, China 5Institute of Advanced Technology, Westlake Institute for Advanced Study, Hangzhou, China {weixiangpeng,huyue}@iie.ac.cn yue.zhang@wias.org.cn {yuheng.yh,wengrx,weihua.luowh}@alibaba-inc.com Abstract Recent evidence reveals that Neural Machine Translation (NMT) models with deeper neu- ral networks can be more effective but are difficult to train. In this paper, we present a MultiScale Collaborative (MSC) framework to ease the training of NMT models that are substantially deeper than those used previ- ously. We explicitly boost the gradient back- propagation from top to bottom levels by in- troducing a block-scale collaboration mecha- nism into deep NMT models. Then, instead of forcing the whole encoder stack directly learns a desired representation, we let each encoder block learns a fine-grained representation and enhance it by encoding spatial dependencies using a context-scale collaboration. We pro- vide empirical evidence showing that the MSC nets are easy to optimize and can obtain im- provements of translation quality from con- siderably increased depth. On IWSLT trans- lation tasks with three translation directions, our extremely deep models (with 72-layer en- coders) surpass strong baselines by +2.2∼+3.1 BLEU points. In addition, our deep MSC achieves a BLEU score of 30.56 on WMT14 English→German task that significantly out- performs state-of-the-art deep NMT models. 1 Introduction Neural machine translation (NMT) directly models the entire translation process using a large neu- ral network and has gained rapid progress in re- cent years (Sutskever et al., 2014; Sennrich et al., 2016). The structure of NMT models has evolved quickly, such as RNN-based (Wu et al., 2016), CNN-based (Gehring et al., 2017) and attention- based (Vaswani et al., 2017) systems. All of these models follow the encoder-decoder frame- work with attention (Cho et al., 2014; Bahdanau et al., 2015; Luong et al., 2015) paradigm. ∗Work done at Alibaba Group. †Corresponding Author. Deep neural networks have revolutionized the state-of-the-art in various communities, from com- puter vision to natural language processing. How- ever, training deep neural networks has been al- ways a challenging problem. To encourage gra- dient flow and error propagation, researchers in the field of computer vision have proposed vari- ous approaches, such as residual connections (He et al., 2016), densely connected networks (Huang et al., 2017) and deep layer aggregation (Yu et al., 2018). In natural language processing, construct- ing deep architectures has shown effectiveness in language modeling, question answering, text clas- sification and natural language inference (Peters et al., 2018; Radford et al., 2018; Al-Rfou et al., 2019; Devlin et al., 2019). However, among ex- isting NMT models, most of them are generally equipped with 4-8 encoder and …

Content truncated to 500 words