Rejuvenating Low-Frequency Words: Making the Most of Parallel Data in Non-Autoregressive Translation

Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing, pages 3431–3441 August 1–6, 2021. ©2021 Association for Computational Linguistics 3431 Rejuvenating Low-Frequency Words: Making the Most of Parallel Data in Non-Autoregressive Translation Liang Ding∗ The University of Sydney ldin3097@sydnye.edu.au Longyue Wang∗ Tencent AI Lab vinnylywang@tencent.com Xuebo Liu University of Macau nlp2ct.xuebo@gmail.com Derek F. Wong University of Macau derekfw@um.edu.com Dacheng Tao JD Explore Academy, JD.com dacheng.tao@gmail.com Zhaopeng Tu Tencent AI Lab zptu@tencent.com Abstract Knowledge distillation (KD) is commonly used to construct synthetic data for training non-autoregressive translation (NAT) models. However, there exists a discrepancy on low- frequency words between the distilled and the original data, leading to more errors on pre- dicting low-frequency words. To alleviate the problem, we directly expose the raw data into NAT by leveraging pretraining. By analyz- ing directed alignments, we found that KD makes low-frequency source words aligned with targets more deterministically but fails to align sufficient low-frequency words from tar- get to source. Accordingly, we propose reverse KD to rejuvenate more alignments for low- frequency target words. To make the most of authentic and synthetic data, we combine these complementary approaches as a new train- ing strategy for further boosting NAT perfor- mance. We conduct experiments on five trans- lation benchmarks over two advanced architec- tures. Results demonstrate that the proposed approach can significantly and universally im- prove translation quality by reducing transla- tion errors on low-frequency words. Encour- agingly, our approach achieves 28.2 and 33.9 BLEU points on the WMT14 English-German and WMT16 Romanian-English datasets, re- spectively. Our code, data, and trained mod- els are available at https://github.com/ longyuewangdcu/RLFW-NAT. 1 Introduction Recent years have seen a surge of interest in non- autoregressive translation (NAT, Gu et al., 2018), which can improve the decoding efficiency by pre- dicting all tokens independently and simultane- ously. The non-autoregressive factorization breaks conditional dependencies among output tokens, ∗ Liang Ding and Longyue Wang contributed equally to this work. Work was done when Liang Ding and Xuebo Liu were interning at Tencent AI Lab. which prevents a model from properly capturing the highly multimodal distribution of target trans- lations. As a result, the translation quality of NAT models often lags behind that of autoregressive translation (AT, Vaswani et al., 2017) models. To balance the trade-off between decoding speed and translation quality, knowledge distillation (KD) is widely used to construct a new training data for NAT models (Gu et al., 2018). Specifically, target sentences in the distilled training data are gener- ated by an AT teacher, which makes NAT easily acquire more deterministic knowledge and achieve significant improvement (Zhou et al., 2020). Previous studies have shown that distillation may lose some important information in the original training data, leading to more errors on predict- ing low-frequency words. To alleviate this prob- lem, Ding et al. (2021b) proposed to augment NAT models the ability to learn lost knowledge from the original data. However, their approach relies on external resources …

Content truncated to 500 words