An analysis of language models for metaphor recognition

Proceedings of the 28th International Conference on Computational Linguistics, pages 3722–3736 Barcelona, Spain (Online), December 8-13, 2020 3722 An analysis of language models for metaphor recognition Arthur Neidlein, Philipp Wiesenbach and Katja Markert Institute of Computational Linguistics Heidelberg University {neidlein|wiesenbach|markert}@cl.uni-heidelberg.de Abstract We conduct a linguistic analysis of recent metaphor recognition systems, all of which are based on language models. We show that their performance, although reaching high F-scores, has considerable gaps from a linguistic perspective. First, they perform substantially worse on un- conventional metaphors than on conventional ones. Second, they struggle with handling rarer word types. These two findings together suggest that a large part of the systems’ success is due to optimising the disambiguation of conventionalised, metaphoric word senses for specific words instead of modelling general properties of metaphors. As a positive result, the systems show increasing capabilities to recognise metaphoric readings of unseen words if synonyms or mor- phological variations of these words have been seen before, leading to enhanced generalisation beyond word sense disambiguation. 1 Introduction Metaphor is a type of figurative language where meaning transfer occurs via similarity between two conceptual domains. In Examples 1 to 3, the metaphors attacked, bashed and ceasefire stem from a transfer from the domain WAR (or FIGHT) to the domain ARGUMENT (Lakoff and Johnson, 1980).1 (1) He attacked my argument. (2) He bashed my argument. (3) We declared a ceasefire during dinner. Metaphoric instances can stem from such regular metaphoric patterns equating two domains habitually or conventionally.2 However, they can also be novel/unconventional such as the famous Emily Dickin- son metaphor Hope is the thing with feathers, which does not correspond to a well-known metaphorical pattern. Even within a metaphorical pattern such as Argument is War, there are degrees of conven- tionality with Example 1 being more conventional than Examples 2 and 3. To a human, unconventional metaphors tend to be more noticeable. Metaphor detection has been studied extensively in NLP in recent years (see (Veale et al., 2016; Shutova et al., 2017) for overviews). State-of-the-art approaches in metaphor detection build strongly on language models and word embeddings, with more than half of the participants in the 2020 Shared Task on Metaphor Detection (Leong et al., 2020) using a variant of BERT language models (Devlin et al., 2019). Evaluations on the standard metaphor recognition test sets report scores that creep up steadily, using such methods. We investigate whether these models really are able to learn general properties of metaphor. To do so and to go beyond word sense disambiguation, they should be able to (i) recognise conventional and unconventional metaphors (ii) be able to perform well on rarer word types that often This work is licensed under a Creative Commons Attribution 4.0 International Licence. Licence details: http:// creativecommons.org/licenses/by/4.0/. 1In our examples, metaphoric words are marked in italics. 2Lakoff and Johnson (1980) call these patterns conceptual metaphors. We will mainly use the term metaphoric patterns to distinguish those clearly from specific metaphoric instances, which we simply call metaphors. 3723 follow the …

Content truncated to 500 words