Disentangling Semantics and Syntax in Sentence Embeddings with Pre-trained Language Models

Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 1372–1379 June 6–11, 2021. ©2021 Association for Computational Linguistics 1372 Disentangling Semantics and Syntax in Sentence Embeddings with Pre-trained Language Models James Y. Huang and Kuan-Hao Huang and Kai-Wei Chang University of California, Los Angeles {jyhuang, khhuang, kwchang}@cs.ucla.edu Abstract Pre-trained language models have achieved huge success on a wide range of NLP tasks. However, contextual representations from pre- trained models contain entangled semantic and syntactic information, and therefore cannot be directly used to derive useful semantic sen- tence embeddings for some tasks. Paraphrase pairs offer an effective way of learning the distinction between semantics and syntax, as they naturally share semantics and often vary in syntax. In this work, we present ParaBART, a semantic sentence embedding model that learns to disentangle semantics and syntax in sentence embeddings obtained by pre-trained language models. ParaBART is trained to per- form syntax-guided paraphrasing, based on a source sentence that shares semantics with the target paraphrase, and a parse tree that speci- fies the target syntax. In this way, ParaBART learns disentangled semantic and syntactic rep- resentations from their respective inputs with separate encoders. Experiments in English show that ParaBART outperforms state-of-the- art sentence embedding models on unsuper- vised semantic similarity tasks. Additionally, we show that our approach can effectively re- move syntactic information from semantic sen- tence embeddings, leading to better robustness against syntactic variation on downstream se- mantic tasks. 1 Introduction Semantic sentence embedding models encode sen- tences into fixed-length vectors based on their se- mantic relatedness with each other. If two sen- tences are more semantically related, their corre- sponding sentence embeddings are closer. As sen- tence embeddings can be used to measures seman- tic relatedness without requiring supervised data, they have been used in many applications, such as semantic textual similarity (Agirre et al., 2016a), question answering (Nakov et al., 2017), and nat- ural language inference (Artetxe and Schwenk, 2019a). Recent years have seen huge success of pre- trained language models across a wide range of NLP tasks (Devlin et al., 2019; Lewis et al., 2020). However, several studies (Reimers and Gurevych, 2019; Li et al., 2020) have found that sentence embeddings from pre-trained language models per- form poorly on semantic similarity tasks when the models are not fine-tuned on task-specific data. Meanwhile, Goldberg (2019) shows that BERT without fine-tuning performs surprisingly well on syntactic tasks. Hence, we posit that these con- textual representations from pre-trained language models without fine-tuning capture entangled se- mantic and syntactic information, and therefore are not suitable for sentence-level semantic tasks. Ideally, the semantic embedding of a sentence should not encode its syntax, and two semantically similar sentences should have close semantic em- beddings regardless of their syntactic differences. While various models (Conneau et al., 2017; Cer et al., 2018; Reimers and Gurevych, 2019) have been proposed to improve the performance of sen- tence embeddings on downstream semantic tasks, most of these approaches do not …

Content truncated to 500 words