Semantic Data Set Construction from Human Clustering and Spatial Arrangement

Semantic Data Set Construction from Human Clustering and Spatial Arrangement Olga Majewska Language Technology Lab University of Cambridge om304@cam.ac.uk Diana McCarthy Language Technology Lab University of Cambridge diana@dianamccarthy.co.uk Jasper J. F. van den Bosch School of Psychology University of Birmingham vandejjf@bham.ac.uk Nikolaus Kriegeskorte Zuckerman Institute University of Columbia nk2765@columbia.edu Ivan Vuli´c Language Technology Lab University of Cambridge iv250@cam.ac.uk Anna Korhonen Language Technology Lab University of Cambridge alk23@cam.ac.uk Research into representation learning models of lexical semantics usually utilizes some form of intrinsic evaluation to ensure that the learned representations reflect human semantic judg- ments. Lexical semantic similarity estimation is a widely used evaluation method, but efforts Submission received: 20 April 2020; revised version received: 23 August 2020; accepted for publication: 3 December 2020. https://doi.org/10.1162/COLI a 00396 © 2021 Association for Computational Linguistics Published under a Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 International (CC BY-NC-ND 4.0) license Computational Linguistics Volume 47, Number 1 have typically focused on pairwise judgments of words in isolation, or are limited to specific contexts and lexical stimuli. There are limitations with these approaches that either do not provide any context for judgments, and thereby ignore ambiguity, or provide very specific sentential contexts that cannot then be used to generate a larger lexical resource. Furthermore, similarity between more than two items is not considered. We provide a full description and analysis of our recently proposed methodology for large-scale data set construction that pro- duces a semantic classification of a large sample of verbs in the first phase, as well as multi- way similarity judgments made within the resultant semantic classes in the second phase. The methodology uses a spatial multi-arrangement approach proposed in the field of cognitive neuroscience for capturing multi-way similarity judgments of visual stimuli. We have adapted this method to handle polysemous linguistic stimuli and much larger samples than previous work. We specifically target verbs, but the method can equally be applied to other parts of speech. We perform cluster analysis on the data from the first phase and demonstrate how this might be useful in the construction of a comprehensive verb resource. We also analyze the semantic information captured by the second phase and discuss the potential of the spatially induced similarity judgments to better reflect human notions of word similarity. We demonstrate how the resultant data set can be used for fine-grained analyses and evaluation of representation learning models on the intrinsic tasks of semantic clustering and semantic similarity. In particular, we find that stronger static word embedding methods still outperform lexical representations emerging from more recent pre-training methods, both on word-level similarity and clustering. Moreover, thanks to the data set’s vast coverage, we are able to compare the benefits of specializing vector representations for a particular type of external knowledge by evaluating FrameNet- and VerbNet-retrofitted models on specific semantic domains such as “Heat” or “Motion.” 1. Introduction Recent advances in representation learning have transformed the NLP landscape, intro- ducing new powerful architectures that achieve unprecedented results on a plethora of natural language tasks (Peters et …

Content truncated to 500 words