LICHEE: Improving Language Model Pre-training with Multi-grained Tokenization

Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021, pages 120–133 August 1–6, 2021. ©2021 Association for Computational Linguistics 120 Fully Non-autoregressive Neural Machine Translation: Tricks of the Trade Jiatao Gu∗ Facebook AI Research jgu@fb.com Xiang Kong∗ Language Technologies Institute Carnegie Mellon University xiangk@cs.cmu.edu Abstract Fully non-autoregressive neural machine trans- lation (NAT) simultaneously predicts tokens with single forward of neural networks, which significantly reduces the inference latency at the expense of quality drop compared to the Transformer baseline. In this work, we target on closing the performance gap while main- taining the latency advantage. We first inspect the fundamental issues of fully NAT models, and adopt dependency reduction in the learn- ing space of output tokens as the primary guid- ance. Then, we revisit methods in four dif- ferent aspects that have been proven effective for improving NAT models, and carefully com- bine these techniques with necessary modifi- cations. Our extensive experiments on three translation benchmarks show that the proposed system achieves the state-of-the-art results for fully NAT models, and obtains comparable performance with the autoregressive and itera- tive NAT systems. For instance, one of the pro- posed models achieves 27.49 BLEU points on WMT14 En-De with 16.5× speed-up com- pared to similar sized autoregressive baseline under the same inference condition. The im- plementation of our model is available here1. 1 Introduction State-of-the-art neural machine translation (NMT) systems are based on autoregressive models (Bah- danau et al., 2015; Vaswani et al., 2017) where each generation step depends on the previously gener- ated tokens. This sequential nature inevitably leads to inherent latency at inference time. On the other hand, non-autoregressive neural machine transla- tion models (NAT, Gu et al., 2018a) attempt to generate output sequences in parallel to speed-up ∗ Equal contribution. 1https://github.com/pytorch/fairseq/ tree/master/examples/nonautoregressive_ translation Figure 1: The translation quality v.s. inference speed- up on WMT’14 En→De test set. The upper right corner achieves the best trade-off. the decoding process. The incorrect independence assumption nevertheless prevents NAT models to properly learn the dependency between target to- kens in real data distribution, resulting in poorer performance compared to autoregressive (AT) mod- els. One popular solution to improve the NAT trans- lation accuracy is to sacrifice the speed-up by incor- porating an iterative refinement process, through which the model explicitly learns the conditional distribution over partially observed reference to- kens (Ghazvininejad et al., 2019; Gu et al., 2019). However, recent studies (Kasai et al., 2020b) indi- cated that iterative NAT models seem to lose the speed advantage compared to AT models with care- ful tuning of the layer allocation. For instance, an AT model with deep encoder and shallow decoder obtains similar latency as iterative NAT models without hurting the translation accuracy. Therefore, how to build a competitive fully NAT model without iterative refinements calls for more exploration. Several works (Ghazvininejad et al., 2020a; Saharia et al., 2020; Qian et al., 2020) have recently been proposed to improve the training of NAT, though the performance gap compared to the iterative ones remains. In …

Content truncated to 500 words