An Evaluation of Disentangled Representation Learning for Texts

Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021, pages 1939–1951 August 1–6, 2021. ©2021 Association for Computational Linguistics 1939 An Evaluation of Disentangled Representation Learning for Texts Krishnapriya Vishnubhotla1,2, Graeme Hirst1, and Frank Rudzicz1,2,3 1Department of Computer Science, University of Toronto 2Vector Institute for Artificial Intelligence 3Unity Health Toronto {vkpriya,gh,frank}@cs.toronto.edu Abstract Learning disentangled representations of texts, which encode information pertaining to differ- ent aspects of the text in separate representa- tions, is an active area of research in NLP for controllable and interpretable text generation. These methods have, for the most part, been developed in the context of text style transfer, but are limited in their evaluation. In this work, we look at the motivation behind learning dis- entangled representations of content and style for texts and at the potential use-cases when compared to end-to-end methods. We then propose evaluation metrics that correspond to these use-cases. We conduct a systematic in- vestigation of previously proposed loss func- tions for such models and we evaluate them on a highly-structured and synthetic natural lan- guage dataset that is well-suited for the task of disentangled representation learning, as well as two other parallel style transfer datasets. Our results demonstrate that current models still require considerable amounts of supervi- sion in order to achieve good performance. 1 Introduction The similarity of texts can be assessed along mul- tiple dimensions. They could contain the same topics, as identified by semantic similarity. They could belong to the same genre or be written by the same author, in which case we might identify stylistic similarity. Texts that present a positive sen- timent may be considered similar to one another when compared to those that express a negative sen- timent, even if they talk about different topics. The similarity of texts, therefore, must be defined to- gether with a frame of reference or a pre-specified dimension of variation. Text representations obtained by current repre- sentation learning methods combine all of these dif- ferent aspects of a text into a single vector embed- ding (Conneau et al., 2017; Reimers and Gurevych, 2019). This results in only a fuzzy measure of text similarity when it is calculated using methods such as the cosine distance between vector embed- dings. Recently, some research in NLP has focused on learning disentangled representations for texts, which aim to capture the different dimensions of variation of a text in separate vector embeddings. These methods have been investigated for style transfer to obtain disentangled representations of content and style (John et al., 2019; Romanov et al., 2019; Cheng et al., 2020), and paraphrase genera- tion for disentangling syntax and semantics (Chen et al., 2019; Balasubramanian et al., 2020). In- spired by parallel developments on style transfer and disentanglement in computer vision, many of them operate within the variational autoencoder framework, where the autoencoder is modified to now encode a text into two latent vectors: one cap- turing the style (the aspect of variation), and the other capturing the content. Style transfer is then achieved by …

Content truncated to 500 words