A Systematic Search for Compound Semantics in Pretrained BERT Architectures

Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics, pages 1499–1512 May 2-6, 2023 ©2023 Association for Computational Linguistics A Systematic Search for Compound Semantics in Pretrained BERT Architectures Filip Miletic and Sabine Schulte im Walde Institute for Natural Language Processing, University of Stuttgart {filip.miletic, schulte}@ims.uni-stuttgart.de Abstract To date, transformer-based models such as BERT have been less successful in predict- ing compositionality of noun compounds than static word embeddings. This is likely related to a suboptimal use of the encoded information, reflecting an incomplete grasp of how the mod- els represent the meanings of complex linguis- tic structures. This paper investigates variants of semantic knowledge derived from pretrained BERT when predicting the degrees of composi- tionality for 280 English noun compounds as- sociated with human compositionality ratings. Our performance strongly improves on earlier unsupervised implementations of pretrained BERT and highlights beneficial decisions in data preprocessing, embedding computation, and compositionality estimation. The distinct linguistic roles of heads and modifiers are re- flected by differences in BERT-derived repre- sentations, with empirical properties such as frequency, productivity, and ambiguity affect- ing model performance. The most relevant rep- resentational information is concentrated in the initial layers of the model architecture. 1 Introduction The meaning of multiword expressions such as noun compounds is notoriously difficult to model, particularly because of variability in their degree of compositionality, i.e. the relatedness of the meaning of a compound (e.g. flea market) to that of the individual constituents (flea and market). The degree of compositionality has been success- fully predicted using static word embeddings, but transformer-based models such as BERT (Devlin et al., 2019) have so far been less successful. This might be related to a suboptimal use of the infor- mation encoded by the models, reflecting our in- complete grasp of how they represent the meanings of complex linguistic structures. In this paper, we aim to improve this understand- ing, as well as produce actionable methodological recommendations. We predict the degrees of com- positionality of 280 English noun compounds as- sociated with human compositionality ratings. We extract their occurrences from a web corpus and model them using pretrained BERT. Like previous work, we assume that the contextualized nature of these representations may capture key aspects of compound semantics. But we do not expect this information to be equally accessible across the model or independent from underlying linguistic properties. We therefore experiment with variants of BERT-derived semantic knowledge (comprising over 40,000 ways of computing the degree of com- positionality), and analyze the linguistic roles of compound constituents and their empirical proper- ties (frequency, ambiguity, and productivity). We provide the following contributions: • We identify a robust setup to extract compo- sitionality information from pretrained BERT. It strongly improves on earlier unsupervised implementations and highlights beneficial de- cisions in data preprocessing, embedding com- putation, and compositionality estimation. • We show that the distinct linguistic roles of heads and modifiers are reflected by differ- ences in BERT-derived representations. Fur- ther focusing on compound heads, we …

Content truncated to 500 words