Disentangled Representation Learning for Non-Parallel Text Style Transfer

Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 424–434 Florence, Italy, July 28 - August 2, 2019. c⃝2019 Association for Computational Linguistics 424 Disentangled Representation Learning for Non-Parallel Text Style Transfer Vineet John, Lili Mou, Hareesh Bahuleyan, Olga Vechtomova University of Waterloo {vineet.john,hpallika,ovechtom}@uwaterloo.ca doublepower.mou@gmail.com Abstract This paper tackles the problem of disentan- gling the latent representations of style and content in language models. We propose a simple yet effective approach, which incorpo- rates auxiliary multi-task and adversarial ob- jectives, for style prediction and bag-of-words prediction, respectively. We show, both qual- itatively and quantitatively, that the style and content are indeed disentangled in the latent space. This disentangled latent representation learning can be applied to style transfer on non-parallel corpora. We achieve high perfor- mance in terms of transfer accuracy, content preservation, and language fluency, in compar- ison to various previous approaches.1 1 Introduction The neural network has been a successful learning machine during the past decade due to its highly expressive modeling capability, which is a conse- quence of multiple layers of non-linear transfor- mations of input features. Such transformations, however, make intermediate features “latent,” in the sense that they do not have explicit meaning and are not interpretable. Therefore, neural net- works are usually treated as black-box machinery. Disentangling the latent space of neural net- works has become an increasingly important re- search topic. In the image domain, for example, Chen et al. (2016) use adversarial and information maximization objectives to produce interpretable latent representations that can be tweaked to ad- just writing style for handwritten digits, as well as lighting and orientation for face models. However, this problem is less explored in natural language processing. 1Our code and all model output are avail- able at https://sites.google.com/view/ disentangle4transfer. In this paper, we address the problem of dis- entangling the latent space of neural networks for text generation. Our model is built on an autoen- coder that encodes a sentence to the latent space (vector representation) by learning to reconstruct the sentence itself. We would like the latent space to be disentangled with respect to different fea- tures, namely, style and content in our task. To accomplish this, we propose a simple yet ef- fective approach that combines multi-task and ad- versarial objectives. We artificially divide the la- tent representation into two parts: the style space and content space, where we consider the senti- ment of a sentence as its style. We design a sys- tematic set of auxiliary losses, enforcing the sepa- ration of style and content latent spaces. In partic- ular, the multi-task loss operates on a latent space to ensure that the space does contain the infor- mation we wish to encode. The adversarial loss, on the contrary, minimizes the predictability of in- formation that should not be contained in a given latent space. In early work, researchers typically work with the style space (Shen et al., 2017; Fu et al., 2018), but simply ignore the content space, as it is hard …

Content truncated to 500 words