Cyclical Annealing Schedule: A Simple Approach to Mitigating KL Vanishing

Proceedings of NAACL-HLT 2019, pages 240–250 Minneapolis, Minnesota, June 2 - June 7, 2019. c⃝2019 Association for Computational Linguistics 240 Cyclical Annealing Schedule: A Simple Approach to Mitigating KL Vanishing Hao Fu1†, Chunyuan Li2†⇤, Xiaodong Liu2 Jianfeng Gao2, Asli Celikyilmaz2, Lawrence Carin1 1Duke University 2Microsoft Research, Redmond {chunyl, xiaodl, jfgao, aslicel}@microsoft.com {hao.fu, lcarin}@duke.edu Abstract Variational autoencoders (VAEs) with an auto- regressive decoder have been applied for many natural language processing (NLP) tasks. The VAE objective consists of two terms, (i) re- construction and (ii) KL regularization, bal- anced by a weighting hyper-parameter β. One notorious training difficulty is that the KL term tends to vanish. In this paper we study scheduling schemes for β, and show that KL vanishing is caused by the lack of good latent codes in training the decoder at the beginning of optimization. To remedy this, we propose a cyclical annealing schedule, which repeats the process of increasing β multiple times. This new procedure allows the progressive learning of more meaningful latent codes, by leverag- ing the informative representations of previ- ous cycles as warm re-starts. The effectiveness of cyclical annealing is validated on a broad range of NLP tasks, including language mod- eling, dialog response generation and unsuper- vised language pre-training. 1 Introduction Variational autoencoders (VAEs) (Kingma and Welling, 2013; Rezende et al., 2014) have been ap- plied in many NLP tasks, including language mod- eling (Bowman et al., 2015; Miao et al., 2016), dialog response generation (Zhao et al., 2017; Wen et al., 2017), semi-supervised text classifi- cation (Xu et al., 2017), controllable text genera- tion (Hu et al., 2017), and text compression (Miao and Blunsom, 2016). A prominent component of a VAE is the distribution-based latent representation for text sequence observations. This flexible repre- sentation allows the VAE to explicitly model holis- tic properties of sentences, such as style, topic, and high-level linguistic and semantic features. Sam- ples from the prior latent distribution can produce ⇤Corresponding author †Equal Contribution diverse and well-formed sentences through simple deterministic decoding (Bowman et al., 2015). Due to the sequential nature of text, an auto- regressive decoder is typically employed in the VAE. This is often implemented with a recurrent neural network (RNN); the long short-term mem- ory (LSTM) (Hochreiter and Schmidhuber, 1997) RNN is used widely. This introduces one notori- ous issue when a VAE is trained using traditional methods: the decoder ignores the latent variable, yielding what is termed the KL vanishing problem. Several attempts have been made to ameliorate this issue (Yang et al., 2017; Dieng et al., 2018; Zhao et al., 2017; Kim et al., 2018). Among them, perhaps the simplest solution is monotonic KL an- nealing, where the weight of the KL penalty term is scheduled to gradually increase during train- ing (Bowman et al., 2015). While these techniques can effectively alleviate the KL-vanishing issue, a proper unified theoretical interpretation is still lacking, even for the simple annealing scheme. In this paper, we analyze the variable depen- dency in a VAE, and point …

Content truncated to 500 words