Multi-Task Learning with Shared Encoder for Non-Autoregressive Machine Translation

Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 3989–3996 June 6–11, 2021. ©2021 Association for Computational Linguistics 3989 Multi-Task Learning with Shared Encoder for Non-Autoregressive Machine Translation Yongchang Hao†∗ Shilin He‡∗ Wenxiang Jiao‡ Zhaopeng Tu†† Michael R. Lyu‡ Xing Wang†† †School of Computer Science and Technology, Soochow University ‡Department of Computer Science and Engineering, The Chinese University of Hong Kong ††Tencent AI Lab †yongchanghao.w@gmai.com ‡{slhe,wxjiao,lyu}@cse.cuhk.edu.hk ††{zptu,brightxwang}@tencent.com Abstract Non-Autoregressive machine Translation (NAT) models have demonstrated significant inference speedup but suffer from inferior translation accuracy. The common prac- tice to tackle the problem is transferring the Autoregressive machine Translation (AT) knowledge to NAT models, e.g., with knowledge distillation. In this work, we hypothesize and empirically verify that AT and NAT encoders capture different linguistic properties of source sentences. Therefore, we propose to adopt multi-task learning to transfer the AT knowledge to NAT models through encoder sharing. Specifically, we take the AT model as an auxiliary task to enhance NAT model performance. Experi- mental results on WMT14 English⇔German and WMT16 English⇔Romanian datasets show that the proposed MULTI-TASK NAT achieves significant improvements over the baseline NAT models. Furthermore, the performance on large-scale WMT19 and WMT20 English⇔German datasets confirm the consistency of our proposed method. In addition, experimental results demonstrate that our MULTI-TASK NAT is complemen- tary to knowledge distillation, the standard knowledge transfer method for NAT. 1 1 Introduction Neural machine translation (NMT), as the state-of- the-art machine translation paradigm, has recently been approached with two different sequence de- coding strategies. The first type autoregressive ∗The first two authors contributed equally to this work. This work was conducted when Yongchang Hao, Shilin He, and Wenxiang Jiao were interning at Tencent AI Lab. 1 Code is publicly available at https://github. com/yongchanghao/multi-task-nat translation (AT) models generate output tokens one by one following the left to right direction (Vaswani et al., 2017; Bahdanau et al., 2015), but it is often criticized for its slow inference speed (Gu et al., 2018). The second type non-autoregressive transla- tion (NAT) models adopt a parallel decoding algo- rithm to produce output tokens simultaneously (Gu et al., 2019; Ghazvininejad et al., 2019; Ma et al., 2020), but the translation quality of it is often infe- rior to auto-regressive models (Gu et al., 2018). Many researchers have investigated the collabo- ration between AT and NAT models. For instance, ENCODER-NAD-AD (Zhou et al., 2020) leverages NAT models to improve the performance of AT. Specifically, their method inserts a NAT decoder between the conventional AT encoder and decoder to generate coarse target sequences for the final autoregressive decoding. A line of research (Wang et al., 2019b; Guo et al., 2020; Ding et al., 2020) holds the opinion that the lack of contextual de- pendency on target sentences potentially leads to the deteriorated performance of NAT models. To boost the NAT translation performance, many re- cent works resort to the knowledge transfer from a well-trained AT model. Typical knowledge transfer methods include sequence-level knowledge …

Content truncated to 500 words