Does BERT Understand Idioms? A Probing-Based Empirical Study of BERT Encodings of Idioms

Proceedings of Recent Advances in Natural Language Processing, pages 1397–1407 Sep 1–3, 2021. https://doi.org/10.26615/978-954-452-072-4_156 1397 Does BERT Understand Idioms? A Probing-Based Empirical Study of BERT Encodings of Idioms Minghuan Tan and Jing Jiang School of Computing and Information Systems Singapore Management University mhtan.2017@phdcs.smu.edu.sg jingjiang@smu.edu.sg Abstract Understanding idioms is important in NLP. In this paper, we study to what extent a pre- trained BERT model is able to encode the meaning of a potentially idiomatic expres- sion (PIE) in a certain context. We make use of a few existing datasets and perform two prob- ing tasks: PIE usage classification and idiom paraphrase identification. Our experiment re- sults suggest that BERT indeed is able to sep- arate the literal and idiomatic usages of a PIE with high accuracy. It is also able to encode the idiomatic meaning of a PIE to some extent. 1 Introduction Understanding idiomatic expressions is important for NLP tasks such as sentiment analysis (Balahur et al., 2010; Williams et al., 2015) and machine translation (Isabelle et al., 2017; Shao et al., 2018). However, due to the non-compositionality of id- ioms, it remains a challenge to model the semantic meanings of idioms effectively (Sag et al., 2002; Shwartz and Dagan, 2019). BERT is a contextualized pre-trained language model that has been widely used and proven to be highly effective for many NLP tasks (Devlin et al., 2019). To better understand how BERT works, re- cently the community has adopted the approach of probing, where a probing task is designed to test whether BERT encodings contain sufficient in- formation to perform the task well. Examples of probing tasks include POS tagging and parsing (He- witt and Liang, 2019; Wu et al., 2020) as well as semantic reasoning tasks such as understanding numbers (Wallace et al., 2019). It is therefore also natural to ask whether BERT encodes any knowledge about the usage and mean- ings of idioms, given that BERT was trained on huge corpora, which must contain many idiomatic expressions. However, this problem has not been well explored. To the best of our knowledge, the closest existing work is by Shwartz and Dagan (2019), who studied whether pre-trained (static and contextualized) word embeddings can detect mean- ing shift and implicit information of phrases, with the help of several probing tasks. However, we believe there is a need for further exploration. We note that Shwartz and Dagan (2019) did not specif- ically focus on idioms; only one of the six prob- ing tasks was directly related to idioms, and only idiomatic noun compounds were studied. Since English idioms have different syntactic structures, it would be useful to experiment with a higher cov- erage of different types of idioms. In this paper, we focus on probing BERT to un- derstand whether BERT embeddings can encode the meanings of a diverse range of different types of idioms. We propose two probing tasks to test whether BERT understands idioms. First, given a context containing a potentially idiomatic expres- sion (PIE), the task is to …

Content truncated to 500 words