A Universal Discriminator for Zero-Shot Generalization

Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics Volume 1: Long Papers, pages 1018–1032 July 9-14, 2023 ©2023 Association for Computational Linguistics Does GPT-3 Grasp Metaphors? Identifying Metaphor Mappings with Generative Language Models Lennart Wachowiak King’s College London lennart.wachowiak@gmail.com Dagmar Gromann University of Vienna dagmar.gromann@gmail.com Abstract Conceptual metaphors present a powerful cog- nitive vehicle to transfer knowledge structures from a source to a target domain. Prior neural approaches focus on detecting whether natu- ral language sequences are metaphoric or lit- eral. We believe that to truly probe metaphoric knowledge in pre-trained language models, their capability to detect this transfer should be investigated. To this end, this paper proposes to probe the ability of GPT-3 to detect metaphoric language and predict the metaphor’s source do- main without any pre-set domains. We experi- ment with different training sample configura- tions for fine-tuning and few-shot prompting on two distinct datasets. When provided 12 few- shot samples in the prompt, GPT-3 generates the correct source domain for a new sample with an accuracy of 65.15% in English and 34.65% in Spanish. GPT’s most common er- ror is a hallucinated source domain for which no indicator is present in the sentence. Other common errors include identifying a sequence as literal even though a metaphor is present and predicting the wrong source domain based on specific words in the sequence that are not metaphorically related to the target domain. 1 Introduction Metaphor processing with pre-trained language models (e.g. Conneau et al., 2020; Brown et al., 2020) has been dominated by metaphor detec- tion, that is, the classification of expressions into metaphoric or literal (e.g. Aghazadeh et al., 2022; Leong et al., 2020). In metaphor interpretation, a common approach is to paraphrase metaphoric ex- pressions into literal ones (e.g. Stowe et al., 2021a). Few approaches target metaphor identification, e.g. predicting the source domain of a metaphor in a lin- guistic sequence. For instance, Rosen (2018) relies on grammatical constructs and pre-defined labels. Instead, in this paper, we test a generative language model’s ability to predict the source domain given a target domain and sequence without grammatical assumptions or fixed source domain labels. Conceptual metaphor theory (CMT) (Lakoff and Johnson, 1980) starts from the assumption that metaphors represent a powerful cognitive mech- anism to transfer physical knowledge structures to abstract domains. In natural language, He was bombarded by insults or Your words pierce my heart transfers the concrete domain of weapons to the abstract domain of words in the metaphor WORDS ARE WEAPONS. On the assumption that our cognitive organization relies on metaphors, au- tomatically identifying metaphoric transfer holds the promise of contributing to more human-like computational models. From the overall success of pre-trained language models in metaphor detection, a certain degree of metaphoric knowledge in these models can be assumed (Aghazadeh et al., 2022). This paper aims to evaluate whether this inherent knowledge extends beyond contextual clues to pre- dict the concrete domains in the metaphoric trans- fer. Detecting a metaphor entails contrasting …

Content truncated to 500 words