SBERT studies Meaning Representations: Decomposing Sentence Embeddings into Explainable Semantic Features

Proceedings of the 2nd Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics and the 12th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 625–638 November 20–23, 2022. ©2022 Association for Computational Linguistics 625 SBERT studies Meaning Representations: Decomposing Sentence Embeddings into Explainable Semantic Features Juri Opitz Dept. of Computational Linguistics Heidelberg University 69120 Heidelberg opitz.sci@gmail.com Anette Frank Dept. of Computational Linguistics Heidelberg University 69120 Heidelberg frank@cl.uni-heidelberg.de Abstract Models based on large-pretrained language models, such as S(entence)BERT, provide ef- fective and efficient sentence embeddings that show high correlation to human similarity rat- ings, but lack interpretability. On the other hand, graph metrics for graph-based meaning representations (e.g., Abstract Meaning Repre- sentation, AMR) can make explicit the seman- tic aspects in which two sentences are similar. However, such metrics tend to be slow, rely on parsers, and do not reach state-of-the-art performance when rating sentence similarity. In this work, we aim at the best of both worlds, by learning to induce Semantically Structured Sentence BERT embeddings (S3BERT). Our S3BERT embeddings are composed of explain- able sub-embeddings that emphasize various semantic sentence features (e.g., semantic roles, negation, or quantification). We show how to i) learn a decomposition of the sentence em- beddings into semantic features, through ap- proximation of a suite of interpretable AMR graph metrics, and how to ii) preserve the over- all power of the neural embeddings by control- ling the decomposition learning process with a second objective that enforces consistency with the similarity ratings of an SBERT teacher model. In our experimental studies, we show that our approach offers interpretability – while fully preserving the effectiveness and efficiency of the neural sentence embeddings. 1 Introduction Abstract Meaning Representation (AMR) repre- sents the meaning of a sentence as a directed, rooted and acyclic graph (Banarescu et al., 2013). It shows events and entities referred to in a sentence, their semantic roles and key semantic relations such as cause, time, purpose, instrument, negation. The explicit representation of meaning in AMR has motivated research into AMR metrics that mea- sure meaning similarity of the underlying sentences. E.g., AMR metrics are used for semantics-focused NLG evaluation (Opitz and Frank, 2021; Manning and Schneider, 2021; Zeidler et al., 2022), a seman- tic search engine (Bonial et al., 2020), comparison of cross-lingual AMR (Uhrig et al., 2021; Wein et al., 2022), and argument similarity (Opitz et al., 2021b). Moreover, fine-grained AMR metrics can assess meaning similarity of semantic sub-aspects that AMR explicitly captures, e.g., semantic roles or negation (Damonte et al., 2017). However, when measuring similarity rating per- formance against human ratings in the typical zero- shot setting on tasks like STS (Baudiš et al., 2016a) or SICK (Marelli et al., 2014), the (untrained) AMR metrics tend to lag behind large models such as SBERT (Reimers and Gurevych, 2019) that com- putes sentence embeddings with a Siamese BERT transformer model (Devlin et al., 2019). Notably, SBERT alleviates the need for end-to- end similarity inference on each sentence pair. In- …

Content truncated to 500 words