BERT is to NLP what AlexNet is to CV: Can Pre-Trained Language Models Identify Analogies?

Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing, pages 3609–3624 August 1–6, 2021. ©2021 Association for Computational Linguistics 3609 BERT is to NLP what AlexNet is to CV: Can Pre-Trained Language Models Identify Analogies? Asahi Ushio, Luis Espinosa-Anke, Steven Schockaert, Jose Camacho-Collados Cardiff NLP, School of Computer Science and Informatics Cardiff University, United Kingdom {UshioA,Espinosa-AnkeL,SchockaertS1,CamachoColladosJ}@cardiff.ac.uk Abstract Analogies play a central role in human com- monsense reasoning. The ability to recognize analogies such as “eye is to seeing what ear is to hearing”, sometimes referred to as analogi- cal proportions, shape how we structure knowl- edge and understand language. Surprisingly, however, the task of identifying such analogies has not yet received much attention in the lan- guage model era. In this paper, we analyze the capabilities of transformer-based language models on this unsupervised task, using bench- marks obtained from educational settings, as well as more commonly used datasets. We find that off-the-shelf language models can identify analogies to a certain extent, but struggle with abstract and complex relations, and results are highly sensitive to model architecture and hy- perparameters. Overall the best results were obtained with GPT-2 and RoBERTa, while configurations using BERT were not able to outperform word embedding models. Our re- sults raise important questions for future work about how, and to what extent, pre-trained language models capture knowledge about ab- stract semantic relations.1 1 Introduction One of the most widely discussed properties of word embeddings has been their surprising abil- ity to model certain types of relational similari- ties in terms of word vector differences (Mikolov While the title is probably self-explanatory, this is a small note explaining it. BERT is to NLP what AlexNet is to CV is making an analogy on what the BERT and AlexNet models represented for Natural Language Processing (NLP) and Com- puter Vision (CV), respectively. They both brought a paradigm shift in how research was undertaken in their corresponding disciplines and this is what the analogy refers to. 1Source code and data to reproduce our ex- perimental results are available in the following repository: https://github.com/asahi417/ analogy-language-model Query: word:language Candidates: (1) paint:portrait (2) poetry:rhythm (3) note:music (4) tale:story (5) week:year Table 1: An example analogy task from the SAT dataset. The third candidate is the answer to the query. et al., 2013a; Vylomova et al., 2016; Allen and Hospedales, 2019; Ethayarajh et al., 2019). The underlying assumption is that when “a is to b what c is to d” the word vector differences b − a and d − c are expected to be similar, where we write x for the embedding of a word x. While this assump- tion holds for some types of syntactic relations, for semantic relations this holds to a much more limited degree than was suggested in early work (Linzen, 2016; Schluter, 2018). Moreover, the most commonly used benchmarks have focused on spe- cific and well-defined semantic relations such as “capital of”, rather than …

Content truncated to 500 words