Using Wikidata for Enhancing Compositionality in Pretrained Language Models

Proceedings of Recent Advances in Natural Language Processing, pages 170–178 Varna, Sep 4–6, 2023 https://doi.org/10.26615/978-954-452-092-2_019 170 Using Wikidata for Enhancing Compositionality in Pre-trained Language Models Meriem Beloucif1, Mihir Bansal2, Chris Biemann3 1Uppsala University, 2Carnegie Mellon University, 3Universität Hamburg, 1meriem.beloucif@lingfil.uu.se, 2mihirban@andrew.cmu.edu, 3chris.biemann@uni-hamburg.de Abstract One of the many advantages of pre-trained language models (PLMs) such as BERT and RoBERTa is their flexibility and contextual na- ture. These features give PLMs strong capabil- ities for representing lexical semantics. How- ever, PLMs seem incapable of capturing high- level semantics in terms of compositionality. We show that when augmented with the rele- vant semantic knowledge, PMLs learn to cap- ture a higher degree of lexical compositional- ity. We annotate a large dataset from Wikidata highlighting a type of semantic inference that is easy for humans to understand but difficult for PLMs, like the correlation between age and date of birth. We use this resource for fine- tuning DistilBERT, BERT large and RoBERTa. Our results show that the performance of PLMs against the test data continuously improves when augmented with such a rich resource. Our results are corroborated by a consistent improvement over most GLUE benchmark nat- ural language understanding tasks. 1 Introduction Given their recent success in various natural lan- guage processing (NLP) tasks, there has been in- creasing work on understanding the abilities of pre- trained language models (PLMs) beyond what they can memorize. Having been trained on billions of words, BERT (Devlin et al., 2019) has shown impressive language representation abilities. How- ever, there has not been much work on the degree of knowledge that BERT could infer about different topics from just the lexical information that they are trained on. Therefore, there has been a growing interest in probing PLMs on all kinds of linguistic, syntactic and semantic features (Huang et al., 2021; Beloucif and Biemann, 2021; Huang et al., 2021; Mosbach et al., 2020; Tenney et al., 2019; Peters et al., 2018b,a; Devlin et al., 2019; Radford and Narasimhan, 2018; Broscheit et al., 2022). Figure 1: Multiple inferences are systematic for hu- mans; however, they are much harder for NLP models to capture. Figure 1 shows a few examples of high-level se- mantics relating to compositionality. For instance, when asked questions such as “What’s higher Mt. Everest or Mt. Fuji?´´ or “How tall is Bill Clin- ton?´´, a person would most likely, and naturally think about altitude and height respectively, to accu- rately answer this question. When it comes to rea- soning and inferences between semantic attributes (net worth) and their values (rich), humans can systematically infer between these concepts. The closer semantics in NLP that fits this case is com- positional semantics since we investigate how dif- ferent words in a sentence are linked to other words i.e. net worth being linked to wealth, and altitude is linked to the height of a mountain. In this paper, we create a large dataset from Wiki- data (Vrandeˇci´c and Krötzsch, 2014), where each sentence contains two words that are semantically related. We …

Content truncated to 500 words