Improving Disentangled Text Representation Learning with Information-Theoretic Guidance

Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 7530–7541 July 5 - 10, 2020. c⃝2020 Association for Computational Linguistics 7530 Improving Disentangled Text Representation Learning with Information-Theoretic Guidance Pengyu Cheng1∗, Martin Renqiang Min2, Dinghan Shen3, Christopher Malon2, Yizhe Zhang4, Yitong Li1, Lawrence Carin1 1Duke University 2NEC Labs America 3Microsoft Dynamics 365 AI 4Microsoft Research pengyu.cheng@duke.edu, {renqiang, malon}@nec-labs.com {dishen, yizzhang}@microsoft.com, {yitong.li, lcarin}@duke.edu Abstract Learning disentangled representations of natu- ral language is essential for many NLP tasks, e.g., conditional text generation, style trans- fer, personalized dialogue systems, etc. Sim- ilar problems have been studied extensively for other forms of data, such as images and videos. However, the discrete nature of nat- ural language makes the disentangling of tex- tual representations more challenging (e.g., the manipulation over the data space cannot be easily achieved). Inspired by information the- ory, we propose a novel method that effec- tively manifests disentangled representations of text, without any supervision on seman- tics. A new mutual information upper bound is derived and leveraged to measure depen- dence between style and content. By minimiz- ing this upper bound, the proposed method in- duces style and content embeddings into two independent low-dimensional spaces. Experi- ments on both conditional text generation and text-style transfer demonstrate the high quality of our disentangled representation in terms of content and style preservation. 1 Introduction Disentangled representation learning (DRL), which maps different aspects of data into distinct and in- dependent low-dimensional latent vector spaces, has attracted considerable attention for making deep learning models more interpretable. Through a series of operations such as selecting, combin- ing, and switching, the learned disentangled rep- resentations can be utilized for downstream tasks, such as domain adaptation (Liu et al., 2018), style transfer (Lee et al., 2018), conditional genera- tion (Denton et al., 2017; Burgess et al., 2018), and few-shot learning (Kumar Verma et al., 2018). Although widely used in various domains, such ∗ This work was conducted while the first author was doing an internship at NEC Labs America. as images (Tran et al., 2017; Lee et al., 2018), videos (Yingzhen and Mandt, 2018; Hsieh et al., 2018), and speech (Chou et al., 2018; Zhou et al., 2019), many challenges in DRL have received limited exploration in natural language process- ing (John et al., 2019). To disentangle various attributes of text, two distinct types of embeddings are typically consid- ered: the style embedding and the content embed- ding (John et al., 2019). The content embedding is designed to encapsulate the semantic meaning of a sentence. In contrast, the style embedding should represent desired attributes, such as the sentiment of a review, or the personality associated with a post. Ideally, a disentangled-text-representation model should learn representative embeddings for both style and content. To accomplish this, several strategies have been introduced. Shen et al. (2017) proposed to learn a semantically-meaningful content embedding space by matching the content embedding from two dif- ferent style domains. However, their method re- quires predefined style domains, and thus cannot automatically …

Content truncated to 500 words