Hitachi at SemEval-2020 Task 3: Exploring the Representation Spaces of Transformers for Human Sense Word Similarity

Proceedings of the 14th International Workshop on Semantic Evaluation, pages 286–291 Barcelona, Spain (Online), December 12, 2020. 286 Hitachi at SemEval-2020 Task 3: Exploring the Representation Spaces of Transformers for the Human Perception of Word Similarity Terufumi Morishita∗, Gaku Morio*, Hiroaki Ozaki and Toshinori Miyoshi Hitachi, Ltd. Resarch and Development Group Kokubunji, Tokyo, Japan. {terufumi.morishita.wp, gaku.morio.vn, hiroaki.ozaki.yu, toshinori.miyoshi.pd}@hitachi.com Abstract In this paper, we present our system for SemEval-2020 task 3, Predicting the (Graded) Effect of Context in Word Similarity. Due to the unsupervised nature of the task, we concentrated on inquiring about the similarity measures induced by different layers of different pre-trained Transformer-based language models, which can be good approximations of the human sense of word similarity. Interestingly, our experiments reveal a language-independent characteristic: the middle to upper layers of Transformer-based language models can induce good approximate similarity measures. Finally, our system was ranked 1st on the Slovenian part of Subtask1 and 2nd on the Croatian part of both Subtask1 and Subtask2. 1 Introduction In this paper, we describe our participation in SemEval-2020 task 3: Predicting the (Graded) Effect of Context in Word Similarity (Armendariz et al., 2020). The goal of the task is to understand the effect of contexts on word similarity. The task is composed of two subtasks sharing inputs: we are given a word pair (i.e., two words) and two text snippets (hereafter “contexts”) both including the word pair. For convenience of explanation, we describe Subtask2 first. In Subtask2, we predict two similarity scores between the word pair in the two given contexts. In Subtask1, we predict the difference in the above two similarity scores. More detailed descriptions of the subtasks are give in Section 3. In both subtasks, small labeled data is available for model development. Thus, participants are required to build models in an unsupervised manner. We formulated the tasks as the exploration of similarity measures induced in the hidden layer word representation space of pre-trained Transformer (Vaswani et al., 2017) based language models. Our ex- pectation is that contextualized representation of Transformers could induce context dependent similarity measures that approximate the human perception. Experimental results show that better approximations of word sense similarity can be induced in the middle to upper layers of Transformers for most languages. As a result, our system was ranked 1st on the Slovenian part of Subtask 1 and 2nd on the Croatian part of both Subtask 1 and Subtask 2. 2 Background Capturing the similarity between words has been considered to be one of the fundamental tasks in natural language processing research because it is strongly related to many research fields such as text search, entailment recognition, and information extraction. Recent work has been aimed at predicting the sim- ilarity of a given word pair (Camacho-Collados et al., 2017; Mikolov et al., 2013) and considers the similarity of word meanings that does not consider the effects of their contexts. Word-Sense Disam- biguation (WSD) (Miller et al., 2012; Raganato et al., 2017), which is a lexicographical approach to the representation …

Content truncated to 500 words