A Novel Estimator of Mutual Information for Learning to Disentangle Textual Representations

Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing, pages 6539–6550 August 1–6, 2021. ©2021 Association for Computational Linguistics 6539 A Novel Estimator of Mutual Information for Learning to Disentangle Textual Representations Pierre Colombo†⋆, Pablo Piantanida∗, Chloé Clavel⋆ ⋆Télécom ParisTech, Université Paris Saclay † IBM GBS France ∗Laboratoire des Signaux et Systèmes (L2S), CentraleSupelec CNRS Universite Paris-Saclay pierre.colombo@ibm.com chloe.clavel@telecom-paris.fr pablo.piantanida@centralesupelec.fr Abstract Learning disentangled representations of tex- tual data is essential for many natural language tasks such as fair classification, style transfer and sentence generation, among others. The existent dominant approaches in the context of text data either rely on training an adversary (discriminator) that aims at making attribute values difficult to be inferred from the latent code or rely on minimising variational bounds of the mutual information between latent code and the value attribute. However, the available methods suffer of the impossibility to provide a fine-grained control of the degree (or force) of disentanglement. In contrast to adversar- ial methods, which are remarkably simple, al- though the adversary seems to be performing perfectly well during the training phase, af- ter it is completed a fair amount of informa- tion about the undesired attribute still remains. This paper introduces a novel variational up- per bound to the mutual information between an attribute and the latent code of an encoder. Our bound aims at controlling the approxima- tion error via the Renyi’s divergence, leading to both better disentangled representations and in particular, a precise control of the desirable degree of disentanglement than state-of-the-art methods proposed for textual data. Further- more, it does not suffer from the degeneracy of other losses in multi-class scenarios. We show the superiority of this method on fair classifi- cation and on textual style transfer tasks. Ad- ditionally, we provide new insights illustrat- ing various trade-offs in style transfer when at- tempting to learn disentangled representations and quality of the generated sentence. 1 Introduction Learning disentangled representations hold a cen- tral place to build rich embeddings of high- dimensional data. For a representation to be disen- tangled implies that it factorizes some latent cause or causes of variation as formulated by (Bengio et al., 2013). For example, if there are two causes for the transformations in the data that do not gen- erally happen together and are statistically distin- guishable (e.g., factors occur independently), a maximally disentangled representation is expected to present a sparse structure that separates those causes. Disentangled representations have been shown to be useful for a large variety of data, such as video (Hsieh et al., 2018), image (Sanchez et al., 2019), text (John et al., 2018), audio (Hung et al., 2018), among others, and applied to many different tasks, e.g., robust and fair classification (Elazar and Goldberg, 2018), visual reasoning (van Steenkiste et al., 2019), style transfer (Fu et al., 2017), con- ditional generation (Denton et al., 2017; Burgess et al., 2018), few shot learning (Kumar Verma et al., 2018), among …

Content truncated to 500 words