Measuring Context-Word Biases in Lexical Semantic Datasets

Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 2699 - 2713 December 7-11, 2022 ©2022 Association for Computational Linguistics Measuring Context-Word Biases in Lexical Semantic Datasets Qianchu Liu, Diana McCarthy, Anna Korhonen Language Technology Lab, TAL, University of Cambridge, UK {ql261,alk23}@cam.ac.uk diana@dianamccarthy.co.uk Abstract State-of-the-art pretrained contextualized models (PCM) eg. BERT use tasks such as WiC and WSD to evaluate their word- in-context representations. This inherently assumes that performance in these tasks reflect how well a model represents the coupled word and context semantics. We question this assumption by presenting the first quantitative analysis on the context-word interaction being tested in major contextual lexical semantic tasks. To achieve this, we run probing baselines on masked input, and pro- pose measures to calculate and visualize the degree of context or word biases in existing datasets. The analysis was performed on both models and humans. Our findings demonstrate that models are usually not being tested for word-in-context semantics in the same way as humans are in these tasks, which helps us better understand the model-human gap. Specifically, to PCMs, most existing datasets fall into the extreme ends (the retrieval-based tasks exhibit strong target word bias while WiC-style tasks and WSD show strong context bias); In comparison, humans are less biased and achieve much better performance when both word and context are available than with masked input. We recommend our framework for understanding and controlling these biases for model interpretation and future task design. 1 Introduction Meaning contextualization (i.e., identifying the cor- rect meaning of a target word in linguistic context) is essential for understanding natural language, and has been the focus in many lexical semantic tasks. Pretrained contextualized models (PCMs) have brought large improvements in these tasks in- cluding WSD (Hadiwinoto et al., 2019; Loureiro and Jorge, 2019; Huang et al., 2019; Blevins and Zettlemoyer, 2020), WiC (Pilehvar and Camacho- Collados, 2019; Garí Soler et al., 2019) and entity 0.0 0.2 0.4 0.6 0.8 1.0 1.2 Context Bias 0.0 0.2 0.4 0.6 0.8 1.0 1.2 Target Word Bias WiC WiC-TSV WSD SR Aida Wiki WikiMed COMETA AM2iCo XL-WiC MCL-WiC WiC Aida AM2iCo XL-WiC BERT Human Figure 1: Plotting context and target word biases from BERT (black) and humans (blue) across popu- lar context-aware lexical semantic datasets. The green shade and the yellow shade roughly indicate the areas for high target word bias and high context bias (>0.8). We would ideally want a dataset to lie towards the bot- tom left corner which is bias-free. The dashed red lines indicate 1.0 context (right) and 1.0 target word bias (top), implying a dataset is in effect dealt with by re- lying on context alone or target words alone. linking (EL) (Wu et al., 2020; Broscheit, 2019). These superior performances have been taken as proof that PCMs can successfully model word- in-context semantics. Many studies have investi- gated the process of lexical contextualization in these PCMs. Specifically, Vuli´c et al. (2020); Aina et al. (2019) found language models ‘contextualize’ words in …

Content truncated to 500 words