Testing the Ability of Language Models to Interpret Figurative Language

Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 4437 - 4452 July 10-15, 2022 ©2022 Association for Computational Linguistics Testing the Ability of Language Models to Interpret Figurative Language Emmy Liu, Chenxuan Cui, Kenneth Zheng, Graham Neubig Language Technologies Institute Carnegie Mellon University {mengyan3,cxcui,kzheng2,gneubig}@cs.cmu.edu Abstract Figurative and metaphorical language are com- monplace in discourse, and figurative expres- sions play an important role in communica- tion and cognition. However, figurative lan- guage has been a relatively under-studied area in NLP, and it remains an open question to what extent modern language models can inter- pret nonliteral phrases. To address this ques- tion, we introduce Fig-QA, a Winograd-style nonliteral language understanding task consist- ing of correctly interpreting paired figurative phrases with divergent meanings. We evalu- ate the performance of several state-of-the-art language models on this task, and find that al- though language models achieve performance significantly over chance, they still fall short of human performance, particularly in zero- or few-shot settings. This suggests that further work is needed to improve the nonliteral rea- soning capabilities of language models.1 1 Introduction All our words are but crumbs that fall down from the feast of the mind (Gibran, 1926). When humans read such a metaphorical phrase, how do they inter- pret it? Conceptual metaphors structure our every- day language and are used to map everyday phys- ical experiences and emotions onto abstract con- cepts (Lakoff and Johnson, 1981). They allow us to communicate complex ideas, to emphasize emo- tions, and to make humorous statements (Fussell and Moss, 2008). However, despite relating words in a way that differs from their accepted defini- tion, these phrases are readily interpreted by human listeners, and are common in discourse (Shutova, 2011), occurring on average every three sentences (Mio and Katz, 1996; Fussell and Moss, 2008) The ability to interpret figurative language has been viewed as a bottleneck in natural language un- 1Code and data are available at https://github.com/ nightingal3/Fig-QA, and a leaderboard can be found at https: //explainaboard.inspiredco.ai/leaderboards?dataset=fig_qa derstanding, but it has not been studied as widely as literal language (Shutova, 2011; Tong et al., 2021). Figurative language often relies on shared common- sense or cultural knowledge, and in some cases may be difficult to solve using language statistics. This presents a challenge to language models (LMs), as strong LMs trained only on text may not be able to make sense of the physical world, nor the social or cultural knowledge that language is grounded in (Bender and Koller, 2020; Bisk et al., 2020). Most previous work on figurative language fo- cuses on metaphor detection, where a model is trained to identify the existence of metaphors in text (Tsvetkov et al., 2014; Stowe and Palmer, 2018; Leong et al., 2020), with datasets consisting mostly of conventionalized metaphors and idioms in wide use. However, identifying these common metaphors that already appear often in language may be an easy task for LMs, and not fully test their ability …

Content truncated to 500 words