Exploiting Inductive Bias in Transformers for Unsupervised Disentanglement of Syntax and Semantics with VAEs

Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 5763 - 5776 July 10-15, 2022 ©2022 Association for Computational Linguistics Exploiting Inductive Bias in Transformers for Unsupervised Disentanglement of Syntax and Semantics with VAEs Ghazi Felhi, Joseph Le Roux LIPN Université Sorbonne Paris Nord - CNRS UMR 7030 F-93430, Villetaneuse, France {felhi, leroux}@lipn.fr Djamé Seddah INRIA Paris Paris, France djame.seddah@inria.fr Abstract We propose a generative model for text genera- tion, which exhibits disentangled latent repre- sentations of syntax and semantics. Contrary to previous work, this model does not need syntac- tic information such as constituency parses, or semantic information such as paraphrase pairs. Our model relies solely on the inductive bias found in attention-based architectures such as Transformers. In the attention of Transformers, keys handle information selection while values specify what information is conveyed. Our model, dubbed QKVAE, uses Attention in its decoder to read latent variables where one latent variable infers keys while another infers values. We run experiments on latent representations and experiments on syntax/semantics transfer which show that QKVAE displays clear signs of disentangled syntax and semantics. We also show that our model displays competitive syn- tax transfer capabilities when compared to su- pervised models and that comparable super- vised models need a fairly large amount of data (more than 50K samples) to outperform it on both syntactic and semantic transfer. The code for our experiments is publicly available1. 1 Introduction Disentanglement, a process aimed at obtaining neu- ral representations with identified meaning, is a crucial component of research on interpretability (Rudin et al., 2022). A form of disentanglement that received a lot of interest from the NLP commu- nity is the separation between syntax and semantics in neural representations (Chen et al., 2019; Bao et al., 2019; Zhang et al., 2019; Chen et al., 2020; Huang and Chang, 2021; Huang et al., 2021). Pre- vious works perform disentanglement using para- phrase pairs as information for semantics, and/or constituency parses as information for syntax. The 1github.com/ghazi-f/QKVAE dependence of models on labeled data is known to entail high cost (see Seddah et al., 2020 on syntactic annotation), and to often require new labels to han- dle problems such as concept drift (Lu et al., 2019) and domain adaptation (Farahani et al., 2021). In light of the above, we propose an unsuper- vised model which directs syntax and semantics into different neural representations without se- mantic or syntactic information. In the Trans- former architecture (Vaswani et al., 2017), the attention mechanism is built upon a query from a set Q, which pools values V through keys K. For each query, values are selected according to their matching score computed by the similarity between their corresponding keys and the query. Building on an analogy between the (K, V ) cou- ple and syntactic roles with their lexical realiza- tions (explicited in §4.2) we present QKVAE2, a Transformer-based Variational Autoencoder (VAE; Kingma and Welling, 2014). To build our model, we …

Content truncated to 500 words