How Does Distilled Data Complexity Impact the Quality and Confidence of Non-Autoregressive Machine Translation?

Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021, pages 4392–4400 August 1–6, 2021. ©2021 Association for Computational Linguistics 4392 How Does Distilled Data Complexity Impact the Quality and Confidence of Non-Autoregressive Machine Translation? Weijia Xu∗† Shuming Ma‡ Dongdong Zhang‡ Marine Carpuat† †Department of Computer Science, University of Maryland ‡Microsoft Research Asia {weijia, marine}@cs.umd.edu, {shumma, dozhang}@microsoft.com Abstract While non-autoregressive (NAR) models are showing great promise for machine translation (MT), their use is limited by their dependence on knowledge distillation from au- toregressive models. To address this issue, we seek to understand why distillation is so effec- tive. Prior work suggests that distilled training data is less complex than manual translations. Based on experiments with the Levenshtein Transformer and the Mask-Predict NAR mod- els on the WMT14 German-English task, this paper shows that different types of complexity have different impacts: while reducing lexical diversity and decreasing reordering complex- ity both help NAR learn better alignment between source and target, and thus improve translation quality, lexical diversity is the main reason why distillation increases model confidence, which affects the calibration of different NAR models differently. 1 Introduction and Background When training NAR models for neural machine translation (NMT), sequence-level knowledge dis- tillation (Kim and Rush, 2016) is key to match the translation quality of autoregressive (AR) mod- els (Gu et al., 2018; Lee et al., 2018; Ghazvinine- jad et al., 2019; Gu et al., 2019). Knowl- edge distillation was first proposed to obtain small student models that match the quality of a higher-capacity teacher models (Liang et al., 2008; Hinton et al., 2015). Sequence-level knowledge distillation (SLKD) trains the student model p(y | x) to approximate the teacher distribu- tion q(y | x) by maximizing the following objec- tive: LSEQ-KD = − � y∈Y q(y | x) log p(y | x) ≈ − � y∈Y 1 [y = ˆy] log p(y | x), where Y repre- sents the space of all possible target sequences, and ˆy is the output from running beam search with the teacher model q. ∗Work done during internship at Microsoft Research Asia. However, we do not yet have a clear picture for how SLKD impacts NAR training. Ren et al. (2020) show that SLKD reduces the degree of dependency between target tokens. Gu et al. (2018) hypothe- size that SLKD reduces the number of modes in the output distribution (alternative translations for a source). This hypothesis was supported by exper- iments that use multiway parallel data to simulate the modes (Zhou et al., 2019). Zhou et al. (2019) also investigate the impact of data complexity on NAR translation quality – they generate distilled data of varying complexity with AR models of different capacity and show that higher-capacity NAR models require more complex distilled data to achieve better translation quality. They further show that generating distilled references with mix- ture of experts (Shen et al., 2019) improves NAR translation quality. However, training samples can be complex in different ways, and it remains un- clear how different types of data …

Content truncated to 500 words