Optimus: Organizing Sentences via Pre-trained Modeling of a Latent Space

Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing, pages 4678–4699, November 16–20, 2020. c⃝2020 Association for Computational Linguistics 4678 OPTIMUS: Organizing Sentences via Pre-trained Modeling of a Latent Space Chunyuan Li, Xiang Gao, Yuan Li, Baolin Peng, Xiujun Li, Yizhe Zhang, Jianfeng Gao Microsoft Research, Redmond {chunyl, xiag, v-liyua, bapeng, xiul, yizzhang, jfgao}@microsoft.com Abstract When trained effectively, the Variational Au- toencoder (VAE) (Kingma and Welling, 2013; Bowman et al., 2016) can be both a powerful generative model and an effective representa- tion learning framework for natural language. In this paper, we propose the first large-scale language VAE model OPTIMUS 1. A univer- sal latent embedding space for sentences is first pre-trained on large text corpus, and then fine-tuned for various language generation and understanding tasks. Compared with GPT-2, OPTIMUS enables guided language generation from an abstract level using the latent vectors. Compared with BERT, OPTIMUS can gener- alize better on low-resource language under- standing tasks due to the smooth latent space structure. Extensive experimental results on a wide range of language tasks demonstrate the effectiveness of OPTIMUS. It achieves new state-of-the-art on VAE language model- ing benchmarks. 1 Introduction Pre-trained language models (PLMs) have substan- tially advanced the state-of-the-art across a variety of natural language processing (NLP) tasks (Peters et al., 2018; Devlin et al., 2019; Yang et al., 2019; Radford et al., 2019; Liu et al., 2019; Keskar et al., 2019; Shoeybi et al., 2019). PLMs are often trained to predict words based on their context on massive text data, and the learned models can be fine-tuned to adapt to various downstream tasks. PLMs can generally play two different roles: (i) a generic encoder such as BERT (Devlin et al., 2019) to provide contextualized representations for language understanding tasks, and (ii) a powerful decoder such as GPT-2 (Radford et al., 2019) to generate text sequences in an auto-regressive man- ner. In a bid to combine language understanding 1Organizing sentences via Pre-Trained Modeling of a Universal Space and generation tasks in one unified framework, sev- eral model variants have been proposed, including UniLM (Dong et al., 2019), BART (Lewis et al., 2019), and T5 (Raffel et al., 2019). Although signif- icant performance improvement has been reported on a wide range of NLP tasks, these models lack of explicit modeling of structures in a compact la- tent space, rendering it difficult to control language generation/representation from an abstract level. Variational Autoencoders (VAEs) (Kingma and Welling, 2013; Rezende et al., 2014) provide a tractable method to train latent-variable genera- tive models. In NLP, latent variables may assume the role of higher-level sentence representations, which govern a lower-level word-by-word genera- tion process, thus facilitating controlled text gener- ation (Bowman et al., 2016; Hu et al., 2017). By representing sentences in a low-dimensional latent space, VAEs allow easy manipulation of sentences using the corresponding compact vector represen- tations, such as feature regularization specified by prior distributions, and guided sentence generation with interpretable vector operators. Despite the at- tractive theoretical strengths, …

Content truncated to 500 words