Learning Disentangled Textual Representations via Statistical Measures of Similarity

Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics Volume 1: Long Papers, pages 2614 - 2630 May 22-27, 2022 c⃝2022 Association for Computational Linguistics Learning Disentangled Textual Representations via Statistical Measures of Similarity Pierre Colombo1, Guillaume Staerman2, Nathan Noiry2 3, Pablo Piantanida4 1L2S,CentraleSupelec CNRS Universite Paris-Saclay, 2 LTCI, Telecom Paris, Institut Polytechnique de Paris, 3 althiqa, 4 ILLS, Université McGill - ETS - MILA - CNRS - Université Paris-Saclay - CentraleSupélec 1pierre.colombo@centralesupelec.fr Abstract When working with textual data, a natural ap- plication of disentangled representations is fair classification where the goal is to make pre- dictions without being biased (or influenced) by sensitive attributes that may be present in the data (e.g., age, gender or race). Dominant approaches to disentangle a sensitive attribute from textual representations rely on learning si- multaneously a penalization term that involves either an adversarial loss (e.g., a discrimina- tor) or an information measure (e.g., mutual information). However, these methods require the training of a deep neural network with sev- eral parameter updates for each update of the representation model. As a matter of fact, the resulting nested optimization loop is both time consuming, adding complexity to the optimiza- tion dynamic, and requires a fine hyperparame- ter selection (e.g., learning rates, architecture). In this work, we introduce a family of regular- izers for learning disentangled representations that do not require training. These regulariz- ers are based on statistical measures of similar- ity between the conditional probability distri- butions with respect to the sensitive attributes. Our novel regularizers do not require additional training, are faster and do not involve additional tuning while achieving better results both when combined with pretrained and randomly initial- ized text encoders. 1 Introduction As natural language processing (NLP) systems are taken up in an ever wider array of sectors (e.g., le- gal system (Dale, 2019), insurance (Ly et al., 2020), education (Litman, 2016), healthcare (Basyal et al., 2020)), there are growing concerns about the harm- ful potential of bias in such systems (Leidner and Plachouras, 2017). Recently, a large body of re- search aims at analyzing, understanding and ad- dressing bias in various applications of NLP in- cluding language modelling (Liang et al., 2021), machine translation (Stanovsky et al., 2019), toxic- 40 20 0 20 40 20 10 0 10 20 30 Original T=0 male female (a) 20 10 0 10 20 15 10 5 0 5 10 15 20 25 T=10 (b) 20 15 10 5 0 5 10 15 15 10 5 0 5 10 15 T=1000 (c) Fig. 1: PCA followed by a T-SNE projection of BERT embeddings of the sentences of DIAL corpus after T=0,10,1000 iterations of our framework (based on Sinkhorn divergence). Colors display the sensitive (i.e., binary gender) attribute. ity detection (Dixon et al., 2018) and classification (Elazar and Goldberg, 2018). In NLP, current sys- tems often rely on learning continuous embedding of the input text. Thus, it is crucial to ensure that the learnt continuous representations do not ex- hibit bias …

Content truncated to 500 words