Adversarial Decomposition of Text Representation

Proceedings of NAACL-HLT 2019, pages 815–825 Minneapolis, Minnesota, June 2 - June 7, 2019. c⃝2019 Association for Computational Linguistics 815 Adversarial Decomposition of Text Representation Alexey Romanov, Anna Rumshisky, Anna Rogers and David Donahue Department of Computer Science University of Massachusetts Lowell Lowell, MA 01854 {aromanov,arum,arogers}@cs.uml.edu david donahue@student.uml.edu Abstract In this paper, we present a method for ad- versarial decomposition of text representation. This method can be used to decompose a rep- resentation of an input sentence into several independent vectors, each of them responsi- ble for a specific aspect of the input sentence. We evaluate the proposed method on two case studies: the conversion between different so- cial registers and diachronic language change. We show that the proposed method is capable of fine-grained controlled change of these as- pects of the input sentence. It is also learn- ing a continuous (rather than categorical) rep- resentation of the style of the sentence, which is more linguistically realistic. The model uses adversarial-motivational training and includes a special motivational loss, which acts oppo- site to the discriminator and encourages a bet- ter decomposition. Furthermore, we evaluate the obtained meaning embeddings on a down- stream task of paraphrase detection and show that they significantly outperform the embed- dings of a regular autoencoder. 1 Introduction Despite the recent successes in using neural mod- els for representation learning for natural language text, learning a meaningful representation of input sentences remains an open research problem. A variety of approaches, from sequence-to-sequence models that followed the work of Sutskever et al. (2014) to the more recent proposals (Arora et al., 2017; Nangia et al., 2017; Conneau et al., 2017; Logeswaran and Lee, 2018; Subramanian et al., 2018; Cer et al., 2018) share one common draw- back. Namely, all of them encode the input sen- tence into just one single vector of a fixed size. One way to bypass the limitations of a single vec- tor representation is to use an attention mecha- nism (Bahdanau et al., 2014; Vaswani et al., 2017). We propose to approach this problem differently and design a method for adversarial decomposi- tion of the learned input representation into mul- tiple components. Our method encodes the input sentence into several vectors, where each vector is responsible for a specific aspect of the sentence. In terms of learning different separable com- ponents of input representation, our work most closely relates to the style transfer work, which has been applied to a variety of different as- pects of language, from diachronic language dif- ferences (Xu et al., 2012) to authors’ personali- ties (Lipton et al., 2015) and even sentiment (Hu et al., 2017; Fu et al., 2018). The style trans- fer work effectively relies on the more classical distinction between meaning and form (de Saus- sure, 1959), which accounts for the fact that mul- tiple surface realizations are possible for the same meaning. For simplicity, we will use this termi- nology throughout the rest of the paper. Consider encoding an input sentence into a meaning vector …

Content truncated to 500 words