Learning Disentangled Representations for Natural Language Definitions

Findings of the Association for Computational Linguistics: EACL 2023, pages 1371–1384 May 2-6, 2023 ©2023 Association for Computational Linguistics Learning Disentangled Representations for Natural Language Definitions Danilo S. Carvalho1 Giangiacomo Mercatali1† Yingji Zhang1† Andre Freitas1,2 Department of Computer Science, University of Manchester, United Kingdom1 Idiap Research Institute, Switzerland2 <firstname.lastname>@[postgrad.†]manchester.ac.uk Abstract Disentangling the encodings of neural models is a fundamental aspect for improving inter- pretability, semantic control and downstream task performance in Natural Language Process- ing. Currently, most disentanglement methods are unsupervised or rely on synthetic datasets with known generative factors. We argue that recurrent syntactic and semantic regularities in textual data can be used to provide the models with both structural biases and gener- ative factors. We leverage the semantic struc- tures present in a representative and semanti- cally dense category of sentence types, defi- nitional sentences, for training a Variational Autoencoder to learn disentangled represen- tations. Our experimental results show that the proposed model outperforms unsupervised baselines on several qualitative and quantita- tive benchmarks for disentanglement, and it also improves the results in the downstream task of definition modeling. 1 Introduction Learning disentangled representations is a funda- mental step towards enhancing the interpretability of the encodings in deep generative models, as well as improving their downstream performance and generalization ability. Disentangled represen- tations aim to encode the fundamental structure of the data in a more explicit manner, where in- dependent latent variables are embedded for each generative factor (Bengio et al., 2013). Previous work in machine learning proposed to learn disentangled representations by modify- ing the ELBO objective of the Variational Autoen- coders (VAE) (Kingma and Welling, 2014), within an unsupervised framework (Higgins et al., 2017; Kim and Mnih, 2018; Chen et al., 2018). On the other hand, a more recent line of work claims the benefits of supervision in disentanglement (Lo- catello et al., 2019) and it advocates the importance of designing frameworks able to exploit structures Training english poets who lived in the lake district word embedding (w) z qφ w, r pθ ˆ w, ˆr role embedding (r) Differentia Quality Supertype Differentia Event Event Location Evaluation Qualitative Latent traversals Interpolation TSNE Quantitative Disentanglement metrics Downstream task Definition Modeling Figure 1: Left: Supervision mechanism with defini- tion semantic roles (DSR) encoded in the latent space. The dotted arrow represent the conditional VAE version. Right: Evaluation framework. in the data for introducing inductive biases. In par- allel, disentanglement approaches for NLP have been tackling text style transfer, and evaluating the results with extrinsic metrics, such as style transfer accuracy (Hu et al., 2017; John et al., 2019; Cheng et al., 2020). While style transfer approaches investigate the ability to disentangle and control syntactic factors such as tense and gender, the aspect of understand- ing and disentangling the semantic structure in lan- guage is under-explored, but with recent attempts of separating syntactic and semantic latent spaces showing promising results (Chen et al., 2019; Bao et al., 2019). Furthermore, evaluating disentangle- ment is challenging, because it requires knowledge of generative …

Content truncated to 500 words