Leveraging Three Types of Embeddings from Masked Language Models in Idiom Token Classification

Proceedings of the 11th Joint Conference on Lexical and Computational Semantics, pages 234 - 239 July 14-15, 2022 ©2022 Association for Computational Linguistics Leveraging Three Types of Embeddings from Masked Language Models in Idiom Token Classification Ryosuke Takahashi∗ Ryohei Sasano Koichi Takeda Graduate School of Informatics, Nagoya University ryosuke.takahashi.cs@gmail.com, {sasano,takedasu}@i.nagoya-u.ac.jp Abstract Many linguistic expressions have idiomatic and literal interpretations, and the automatic dis- tinction of these two interpretations has been studied for decades. Recent research has shown that contextualized word embeddings derived from masked language models (MLMs) can give promising results for idiom token classifi- cation. This indicates that contextualized word embedding alone contains information about whether the word is being used in a literal sense or not. However, we believe that more types of information can be derived from MLMs and that leveraging such information can im- prove idiom token classification. In this paper, we leverage three types of embeddings from MLMs; uncontextualized token embeddings and masked token embeddings in addition to the standard contextualized word embeddings and show that the newly added embeddings sig- nificantly improve idiom token classification for both English and Japanese datasets. 1 Introduction Potentially idiomatic phrases are often used both in the idiomatic and literal sense. For example, “blew whistle” in (1) is used in the literal sense, whereas that in (2) is used in the idiomatic sense, that is, the meaning of the phrase has shifted and in this case it means accuse. Deciding whether each occurrence of a potentially idiomatic phrase is a literal or idiomatic usage is an essential process for text understanding. We call this processing idiom token classification following Salton et al. (2016). (1) The referee blew the whistle to end the match. (2) I blew the whistle on government corruption. Recently, contextualized word embeddings have been shown to be useful for word sense disam- biguation (Hadiwinoto et al., 2019). Furthermore, *Ryosuke Takahashi is currently at SB Technology Corp. Shwartz and Dagan (2019) showed that the contex- tualized embeddings including BERT (Devlin et al., 2019) are useful for recognizing meaning shift of words in idioms. However, they only used contex- tualized embeddings, even though comparing them with the standard embeddings of the target word can be beneficial for precise detection of meaning shifts. Thus, in this paper, we propose a method to improved a BERT-based idiom token classifier by leveraging uncontextualized word embeddings. Specifically, we use the token embedding of BERT, which is the uncontextualized embedding that is input to BERT and the same vector as is used for the prediction in the task of masked language model. Our assumption can be explained using (1) and (2) as follows: since “whistle” in (2) is used as a part of an idiomatic phrase, its contextualized embedding differs more from the uncontextualized embedding of “whistle” than in the case of (1). Furthermore, we also leverage the masked token embedding of the target word in BERT, which is generated when the target phrase constituents are masked. This embedding can be considered to …

Content truncated to 500 words