A Multi-Task Approach for Disentangling Syntax and Semantics in Sentence Representations

Proceedings of NAACL-HLT 2019, pages 2453–2464 Minneapolis, Minnesota, June 2 - June 7, 2019. c⃝2019 Association for Computational Linguistics 2453 A Multi-Task Approach for Disentangling Syntax and Semantics in Sentence Representations Mingda Chen Qingming Tang Sam Wiseman Kevin Gimpel Toyota Technological Institute at Chicago, Chicago, IL, 60637, USA {mchen,qmtang,swiseman,kgimpel}@ttic.edu Abstract We propose a generative model for a sen- tence that uses two latent variables, with one intended to represent the syntax of the sen- tence and the other to represent its semantics. We show we can achieve better disentangle- ment between semantic and syntactic repre- sentations by training with multiple losses, in- cluding losses that exploit aligned paraphras- tic sentences and word-order information. We also investigate the effect of moving from bag- of-words to recurrent neural network mod- ules. We evaluate our models as well as sev- eral popular pretrained embeddings on stan- dard semantic similarity tasks and novel syn- tactic similarity tasks. Empirically, we find that the model with the best performing syn- tactic and semantic representations also gives rise to the most disentangled representations.1 1 Introduction As generative latent variable models, especially of the continuous variety (Kingma and Welling, 2014; Goodfellow et al., 2014), have become in- creasingly important in natural language process- ing (Bowman et al., 2016; Gulrajani et al., 2017), there has been increased interest in learning mod- els where the latent representations are disentan- gled (Hu et al., 2017). Much of the recent NLP work on learning disentangled representations of text has focused on disentangling the representa- tion of attributes such as sentiment from the rep- resentation of content, typically in an effort to bet- ter control text generation (Shen et al., 2017; Zhao et al., 2017; Fu et al., 2018). In this work, we instead focus on learning sen- tence representations that disentangle the syntax and the semantics of a sentence. We are more- over interested in disentangling these representa- 1Code and data are available at github.com/ mingdachen/disentangle-semantics-syntax tions not for the purpose of controlling generation, but for the purpose of calculating semantic or syn- tactic similarity between sentences (but not both). To this end, we propose a generative model of a sentence which makes use of both semantic and syntactic latent variables, and we evaluate the in- duced representations on both standard semantic similarity tasks and on several novel syntactic sim- ilarity tasks. We use a deep generative model consisting of von Mises Fisher (vMF) and Gaussian priors on the semantic and syntactic latent variables (respec- tively) and a deep bag-of-words decoder that con- ditions on these latent variables. Following much recent work, we learn this model by optimizing the ELBO with a VAE-like (Kingma and Welling, 2014; Rezende et al., 2014) approach. Our learned semantic representations are eval- uated on the SemEval semantic textual similarity (STS) tasks (Agirre et al., 2012; Cer et al., 2017). Because there has been less work on evaluating syntactic representations of sentences, we propose several new syntactic evaluation tasks, which in- volve predicting the syntactic analysis of …

Content truncated to 500 words