Metaphors in Pre-Trained Language Models: Probing and Generalization Across Datasets and Languages

Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics Volume 1: Long Papers, pages 2037 - 2050 May 22-27, 2022 c⃝2022 Association for Computational Linguistics Metaphors in Pre-Trained Language Models: Probing and Generalization Across Datasets and Languages Ehsan Aghazadeh⋆ Mohsen Fayyaz⋆ Yadollah Yaghoobzadeh School of Electrical and Computer Engineering, College of Engineering, University of Tehran, Tehran, Iran {eaghazade1998, mohsen.fayyaz77, y.yaghoobzadeh}@ut.ac.ir Abstract Human languages are full of metaphorical ex- pressions. Metaphors help people understand the world by connecting new concepts and domains to more familiar ones. Large pre- trained language models (PLMs) are therefore assumed to encode metaphorical knowledge useful for NLP systems. In this paper, we in- vestigate this hypothesis for PLMs, by probing metaphoricity information in their encodings, and by measuring the cross-lingual and cross- dataset generalization of this information. We present studies in multiple metaphor detection datasets and in four languages (i.e., English, Spanish, Russian, and Farsi). Our extensive ex- periments suggest that contextual representa- tions in PLMs do encode metaphorical knowl- edge, and mostly in their middle layers. The knowledge is transferable between languages and datasets, especially when the annotation is consistent across training and testing sets. Our findings give helpful insights for both cogni- tive and NLP scientists. 1 Introduction Pre-trained language models (PLMs) (Peters et al., 2018; Devlin et al., 2019), are now used in almost all NLP applications, e.g., machine translation (Li et al., 2021), question answering (Zhang et al., 2020), dialogue systems (Ni et al., 2021), and sen- timent analysis (Minaee et al., 2020). They have sometimes been referred to as “foundation models” (Bommasani et al., 2021) due to their significant impact on research and industry. Metaphors are important aspects of human lan- guages. In conceptual metaphor theory (CMT) (Lakoff and Johnson, 2008), metaphor is defined as a cognitive phenomenon associating two different concepts or domains. This phenomenon is built in cognition and expressed in language. The creativ- ity and problem solving (i.e., generalization to new ⋆ Equal contribution. Figure 1: An illustration of our probing and gener- alization scenarios for metaphorical knowledge. problems) depend on the analogies and metaphors a cognitive system, like our brain, relies on. Mod- eling metaphors is therefore essential in building human-like computational systems that can relate emerging concepts to the more familiar ones. So far, there has been no comprehensive analysis of whether and how PLMs represent metaphori- cal information. We intuitively assume that PLMs must encode some information about metaphors due to their great performance in metaphor detec- tion and other language processing tasks. Con- firming that experimentally is a question that we address here. Specifically, we aim to know whether generalizable metaphorical knowledge is encoded in PLM representations or not. The outline of our work is presented in Figure 1. We first do probing experiments to answer ques- tions such as: (i) with which accuracies and ex- tractablities do different PLMs encode metaphor- ical knowledge? (ii) how deep is the metaphori- cal knowledge encoded in PLM multi-layer repre- sentations? We take two …

Content truncated to 500 words