Does She Wink or Does She Nod? A Challenging Benchmark for Evaluating Word Understanding of Language Models

Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics, pages 532–538 April 19 - 23, 2021. ©2021 Association for Computational Linguistics 532 Does He Wink or Does He Nod? A Challenging Benchmark for Evaluating Word Understanding of Language Models Lutfi Kerem Senel and Hinrich Sch¨utze Center for Information and Language Processing (CIS), LMU Munich, Germany lksenel@gmail.com, inquiries@cislmu.org Abstract Recent progress in pretraining language mod- els on large corpora has resulted in large per- formance gains on many NLP tasks. These large models acquire linguistic knowledge dur- ing pretraining, which helps to improve per- formance on downstream tasks via fine-tuning. To assess what kind of knowledge is acquired, language models are commonly probed by querying them with ‘fill in the blank’ style cloze questions. Existing probing datasets mainly focus on knowledge about relations between words and entities. We introduce WDLMPro (Word Definition Language Model Probing) to evaluate word understanding di- rectly using dictionary definitions of words. In our experiments, three popular pretrained lan- guage models struggle to match words and their definitions. This indicates that they un- derstand many words poorly and that our new probing task is a difficult challenge that could help guide research on LMs in the future. 1 Introduction Natural language processing (NLP) has advanced drastically in the last decade with the design of larger and more sophisticated models, availabil- ity of larger corpora and increasing computational power. Pretrained word embeddings (Mikolov et al., 2013; Pennington et al., 2014) popularized the use of distributed word representations, which became a fundamental building block for NLP systems. Peters et al. (2018a) introduced LSTM- based deep contextual representations and obtained large performance gains by fine-tuning on tasks af- ter unsupervised pretraining (Radford et al., 2018; Howard and Ruder, 2018). More recently, the at- tention based transformer architecture was shown to use context more effectively (Vaswani et al., 2017) and several subsequent models achieved state of the art results in many NLP tasks by combin- ing the transformer architecture with unsupervised pretraining and task specific fine-tuning (Devlin et al., 2019; Liu et al., 2019). Radford et al. (2019) showed that language models can be applied to a variety of tasks without task specific fine tuning. This is demonstrated on a much larger scale by Brown et al. (2020). Deep models improve performance. However, what they actually learn about language and word meaning is still to a large extent unclear due to their uninterpretable nature. For static word embed- dings, researchers used word similarity (Hill et al., 2015) and word analogy (Gladkova et al., 2016) tests to shed light on what information is captured in these dense vector spaces. For language models, a great amount of linguistic knowledge is stored in the model parameters (Peters et al., 2018b). Several studies proposed using ‘fill in the blank’ type cloze statements to test knowledge learned by these models during unsupervised pretraining. Petroni et al. (2019) proposed the LAMA (LAn- guage Model Analysis) probe to test the factual and …

Content truncated to 500 words