latent-GLAT: Glancing at Latent Variables for Parallel Text Generation

Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics Volume 1: Long Papers, pages 8398 - 8409 May 22-27, 2022 c⃝2022 Association for Computational Linguistics latent-GLAT: Glancing at Latent Variables for Parallel Text Generation Yu Bao♠♦, Hao Zhou♣, Shujian Huang♠♦∗, Dongqi Wang♠♦ Lihua Qian♣, Xinyu Dai♠♦, Jiajun Chen♠♦, Lei Li♥† ♠National Key Laboratory for Novel Software Technology, Nanjing University, China ♦Collaborative Innovation Center of Novel Software Technology and Industrialization, China ♣Bytedance AI Lab, China ♥University of California Santa Barbara, USA {baoy,wangdq}@smail.nju.edu.cn {zhouhao.nlp,qianlihua}@bytedance.com {huangsj,daixinyu,chenjj}@nju.edu.cn leili@cs.ucsb.edu Abstract Recently, parallel text generation has received widespread attention due to its success in gen- eration efficiency. Although many advanced techniques are proposed to improve its gen- eration quality, they still need the help of an autoregressive model for training to overcome the one-to-many multi-modal phenomenon in the dataset, limiting their applications. In this paper, we propose latent-GLAT, which em- ploys the discrete latent variables to capture word categorical information and invoke an advanced curriculum learning technique, alle- viating the multi-modality problem. Exper- iment results show that our method outper- forms strong baselines without the help of an autoregressive model, which further broadens the application scenarios of the parallel decod- ing paradigm. ‡ 1 Introduction Non-autoregressive Transformer (NAT, Gu et al., 2018) introduce a parallel decoding paradigm with higher decoding efficiency (> 10×) than autore- gressive models (Bahdanau et al., 2015; Gehring et al., 2017; Vaswani et al., 2017). Unlike autore- gressive models, NAT models impose conditional independence assumptions in words to support par- allel decoding of sentences during inference. It attracts many researchers to explore NAT in ma- chine translation (Gu et al., 2018; Lee et al., 2018; Kaiser et al., 2018) and text-to-speech tasks (Chen et al., 2019; Peng et al., 2020). Amount of researchers devoted themselves to im- prove the NATs’ inferior generation quality. Such as modeling word inter-dependencies by curricu- lum learning (Guo et al., 2020a; Liu et al., 2020) or iterative refinements mechanism (Ghazvininejad *Shujian Huang is the corresponding author. †Work is done while at ByteDance AI Lab. ‡The implementation of latent-GLAT will be released at https://github.com/baoy-nlp/Latent-GLAT. et al., 2019; Guo et al., 2020b), introducing latent variables to decompose target sentences and serve as the springboard for decoding (Shu et al., 2019; Ma et al., 2019; Bao et al., 2021), and introduce in- ductive bias for models’ training (Wei et al., 2019; Li et al., 2019). The most successful method is the glancing transformer (GLAT, Qian et al., 2021a), which trains the NAT model by sampling partial tar- get words as inputs to predict the remaining target words, explicitly building dependencies between the observed and unobserved words. Qian et al. (2021b) employ GLAT to achieve impressive re- sults on the translation task of WMT211, even out- performing many strong autoregressive translation systems in BLEU score (Papineni et al., 2002). Although existing NAT models achieve competi- tive results compared to autoregressive models in translation tasks, it is not negligible that they still need the help of an autoregressive Transformer …

Content truncated to 500 words