Plug and Play Autoencoders for Conditional Text Generation

Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing, pages 6076–6092, November 16–20, 2020. c⃝2020 Association for Computational Linguistics 6076 Plug and Play Autoencoders for Conditional Text Generation Florian Mai†♠ Nikolaos Pappas♣ Ivan Montero♣ Noah A. Smith♣♦ James Henderson† †Idiap Research Institute, Martigny, Switzerland ♠EPFL, Lausanne, Switzerland ♣University of Washington, Seattle, WA, USA ♦Allen Institute for Artificial Intelligence, Seattle, WA, USA {florian.mai,james.henderson@idiap.ch} {npappas,ivamon,nasmith@cs.washington.edu} Abstract Text autoencoders are commonly used for con- ditional generation tasks such as style transfer. We propose methods which are plug and play, where any pretrained autoencoder can be used, and only require learning a mapping within the autoencoder’s embedding space, training embedding-to-embedding (Emb2Emb). This reduces the need for labeled training data for the task and makes the training procedure more efficient. Crucial to the success of this method is a loss term for keeping the mapped embedding on the manifold of the autoen- coder and a mapping which is trained to nav- igate the manifold by learning offset vectors. Evaluations on style transfer tasks both with and without sequence-to-sequence supervision show that our method performs better than or comparable to strong baselines while being up to four times faster. 1 Introduction Conditional text generation1 encompasses a large number of natural language processing tasks such as text simplification (Nisioi et al., 2017; Zhang and Lapata, 2017), summarization (Rush et al., 2015; Nallapati et al., 2016), machine translation (Bah- danau et al., 2015; Kumar and Tsvetkov, 2019) and style transfer (Shen et al., 2017; Fu et al., 2018). When training data is available, the state of the art includes encoder-decoder models with an atten- tion mechanism (Bahdanau et al., 2015; Vaswani et al., 2017) which are both extensions of the origi- nal sequence-to-sequence framework with a fixed bottleneck introduced by Sutskever et al. (2014). Despite their success, these models are costly to train and require a large amount of parallel data. Yet parallel data is scarce for conditional text generation problems, necessitating unsupervised 1We use this term to refer to text generation conditioned on textual input. Figure 1: The manifold of a text autoencoder is the low- dimensional region of the high-dimensional embedding space where texts are actually embedded. The exam- ple shows the mapping of a source sequence x with embedding zx to zy, which is the embedding of target sequence y such that it reflects the target manifold. solutions. Text autoencoders (Bowman et al., 2016) have proven useful for a particular subclass of un- supervised problems that can be broadly defined as style transfer, i.e., changing the style of a text in such a way that the content of the input is preserved. Examples include sentiment transfer (Shen et al., 2017), sentence compression (Fevry and Phang, 2018), and neural machine translation (Artetxe et al., 2018). Most existing methods specialize autoencoders to the task by conditioning the de- coder on the style attribute of interest (Lample et al., 2019; Logeswaran et al., 2018), assuming the presence of labels during training of the au- toencoder. …

Content truncated to 500 words