How Are Idioms Processed Inside Transformer Language Models?

Proceedings of the The 12th Joint Conference on Lexical and Computational Semantics (*SEM 2023), pages 174–179 July 13-14, 2023 ©2023 Association for Computational Linguistics How Are Idioms Processed Inside Transformer Language Models? Anonymous ACL submission Abstract Idioms such as “call it a day” and “piece of 001 cake”, are ubiquitous in natural language. How 002 do Transformer language models process id- 003 ioms? This study examines this question by 004 analysing three models - BERT, Multilingual 005 BERT, and DistilBERT. We compare the em- 006 beddings of idiomatic and literal expressions 007 across all layers of the networks at both the 008 sentence and word levels. Additionally, we in- 009 vestigate the attention directed from other sen- 010 tence tokens towards a word within an idiom 011 as opposed to in a literal context. Results indi- 012 cate that while the three models exhibit slightly 013 different internal mechanisms, they all repre- 014 sent idioms distinctively compared to literal 015 language, with attention playing a critical role. 016 These findings suggest that idioms are semanti- 017 cally and syntactically idiosyncratic, not only 018 for humans but also for language models. 019 1 Introduction 020 “Why would you put all your eggs in one basket? I 021 can’t wrap my head around it”. Idioms such as “put 022 all one’s eggs in one basket” and “wrap one’s head 023 around” are used frequently in natural conversa- 024 tions. Despite their abundance, much remains to be 025 explored regarding their syntactic, semantic, and 026 pragmatic characteristics, and how they are pro- 027 cessed by the human brain as well as NLP models. 028 Recent Transformer-based large language models 029 have demonstrated strong capabilities in a sweep 030 of tasks involving natural language understanding 031 (e.g. Brown et al. (2020)). However, few attempts 032 have been made to understand the inner workings 033 of these language models in terms of idiom process- 034 ing. In this study, we conduct three experiments to 035 explore the inner workings of transformer language 036 models in idiom processing. Specifically, we inves- 037 tigate the processing of BERT, Multilingual BERT 038 and DistilBERT by comparing the embeddings on 039 the sentence level and on the word level. We also 040 explore the attention mechanism on idioms com- 041 pared to literal contexts. We ask three questions: 042 • How do Transformer language models (LMs) 043 represent idiomatic sentences as opposed to 044 their literal spelt-out counterparts across dif- 045 ferent layers in the network? For example, 046 “Birds of a feather flock together” versus “Peo- 047 ple with similar interests stick together”. 048 • How do LMs represent a word inside an id- 049 iom compared to the same word in a literal 050 context? For example, the word “feather” in 051 “Birds of a feather flock together” versus “My 052 parakeet dropped a green feather.” 053 • How do LMs pay attention to a word inside 054 an idiom compared to a literal context? 055 1.1 …

Content truncated to 500 words