How do Words Contribute to Sentence Semantics? Revisiting Sentence Embeddings with a Perturbation Method

Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics, pages 3001–3010 May 2-6, 2023 ©2023 Association for Computational Linguistics How do Words Contribute to Sentence Semantics? Revisiting Sentence Embeddings with a Perturbation Method Wenlin Yao Lifeng Jin Hongming Zhang Xiaoman Pan Kaiqiang Song Dian Yu Dong Yu Jianshu Chen Tencent AI Lab, Bellevue, WA, USA {wenlinyao,dyu,jianshuchen}@global.tencent.com Abstract Understanding sentence semantics requires an interpretation of the main information from a concrete context. To investigate how individual word contributes to sentence semantics, we pro- pose a perturbation method for unsupervised semantic analysis. We next re-examine SOTA sentence embedding models’ ability to capture the main semantics of a sentence by develop- ing a new evaluation metric to adapt sentence compression datasets for automatic evaluation. Results on three datasets show that unsuper- vised discourse relation recognition can serve as a general inference task that can more effec- tively aggregate information to essential con- tents than several SOTA unsupervised sentence embedding models. 1 1 Introduction Humans are usually able to understand sentence meaning based on a complex cognitive process — composition of words (Löbner, 2013). As main car- riers of information, words generally have different levels of contribution to the final sentence seman- tics. For example, underlined words in “The city of Austin is considering extending downtown parking meter hours to the weekends and later during the week” convey the most important information of this sentence. Therefore, determining the primary semantics (or main meaning) of a sentence and estimating how sentence semantics distributes to individual words play critical roles in understand- ing sentence compositionality. Recently, with the help of large pretrained Transformers, sentence representation learning has achieved great success in downstream NLP tasks (Qiu et al., 2020). However, most work either heav- ily relies on human annotation such as Natural Language Inference (NLI) data (Williams et al., 1Code and data are available at https://github.com/ wenlinyao/EACL23-PrimarySemantics 2018; Bowman et al., 2015) to do fine-tuning (Con- neau et al., 2017a; Reimers and Gurevych, 2019) or adopt unsupervised contrastive learning to learn sentence embeddings (Gao et al., 2021; Chuang et al., 2022). They neglect one critical property of an effective sentence embedding model that es- sential contents should contribute to sentence se- mantics more than non-essential contents when encoding a sentence. We observe that primary semantics can be ac- quired by learning to predict discourse relations because sentence primary semantics usually needs to support the logical relations at the discourse level. It is motivated by the distinction between asserted and projected content in semantic theory (Potts, 2003; Tonhauser et al., 2013; Venhuizen et al., 2018). Asserted content is intended to be presented for discussion and information exchange, whereas projected content represents background information that is not under discussion. Thus, as- serted content should carry more weight in sentence representations than projected content, because it is at issue in the context by the speaker’s intention. Based on this observation, we apply 36 explicit discourse connectives to four big corpora and ex- tract 9.8M …

Content truncated to 500 words