Helping the Weak Makes You Strong: Simple Multi-Task Learning Improves Non-Autoregressive Translators

Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 5513 - 5519 December 7-11, 2022 ©2022 Association for Computational Linguistics Helping the Weak Makes You Strong: Simple Multi-Task Learning Improves Non-Autoregressive Translators Xinyou Wang♠ Zaixiang Zheng♡ Shujian Huang♠ ♠National Key Laboratory for Novel Software Technology, Nanjing University ♡ByteDance AI Lab wangxinyou@smail.nju.edu.cn, zhengzaixiang@bytedance.com huangsj@nju.edu.cn Abstract Recently, non-autoregressive (NAR) neural ma- chine translation models have received increas- ing attention due to their efficient parallel de- coding. However, the probabilistic framework of NAR models necessitates conditional in- dependence assumption on target sequences, falling short of characterizing human language data. This drawback results in less informa- tive learning signals for NAR models under conventional MLE training, thereby yielding unsatisfactory accuracy compared to their au- toregressive (AR) counterparts. In this paper, we propose a simple and model-agnostic multi- task learning framework to provide more in- formative learning signals. During training stage, we introduce a set of sufficiently weak AR decoders that solely rely on the informa- tion provided by NAR decoder to make pre- diction, forcing the NAR decoder to become stronger or else it will be unable to support its weak AR partners. Experiments on WMT and IWSLT datasets show that our approach can consistently improve accuracy of multiple NAR baselines without adding any additional decoding overhead. 1 Introduction State-of-the-art neural machine translation (NMT) systems are mainly autoregressive (AR) mod- els (Bahdanau et al., 2015; Vaswani et al., 2017), which decompose the joint probability of a se- quence of tokens in a left-to-right order, model- ing dependencies of each token with its preceding ones. Despite having strong performance, such sequential decoding causes considerable latency, thereby unsatisfactory efficiency. In contrast, non-autoregressive (NAR) transla- tion models (Gu et al., 2018) permit potentially more efficient parallel decoding. To do so, NAR 1Code will be released at https://github.com/ wxy-nlp/MultiTaskNAT. Encoder NAR Decoder Layer N × x1 x2 x3 x4 y1 Linear+Softmax y2 y3 y4 y5 e1 e2 e3 e4 h1 h2 h3 h4 h5 Self-Attention Cross-Attention Feed Forward Network <bos> y1 y2 y3 y4 Weak AR Decoder y1 y2 y3 y4 y5 Query Key Value forward pass backward pass Figure 1: Illustration of our approach, where we intro- duce a set of auxiliary weak AR decoders, each of which must make its predictions solely relying on the informa- tion contained in the NAR decoder hidden states. Thus, the information provided by the NAR decoder must be sufficiently useful for the AR decoders to be capable of predicting the target sequence because the AR decoders are parameterized as weakly as possible, which will in turn let the NAR decoder learn to get stronger. models necessitate a notorious conditional indepen- dence assumption on target sequences as a trade- off. This assumption, however, is probalistically insufficient to describe the highly multi-modal na- ture of human language data, imposing severe chal- lenges for NAR models in a way of yielding less informative learning signals and gradients under the conventional MLE training. As a result, NAR models often manifest …

Content truncated to 500 words