Predicting human similarity judgments with distributional models: The value of word associations.

Proceedings of COLING 2016, the 26th International Conference on Computational Linguistics: Technical Papers, pages 1861–1870, Osaka, Japan, December 11-17 2016. Predicting human similarity judgments with distributional models: The value of word associations Simon De Deyne and Amy Perfors Computational Cognitive Science Lab School of Psychology University of Adelaide simon.dedeyne@adelaide.edu.au amy.perfors@adelaide.edu.au Daniel J Navarro School of Psychology University of New South Wales dan.navarro@unsw.edu.au Abstract Most distributional lexico-semantic models derive their representations based on external lan- guage resources such as text corpora. In this study, we propose that internal language models, that are more closely aligned to the mental representations of words could provide important in- sights into cognitive science, including linguistics. Doing so allows us to reflect upon theoretical questions regarding the structure of the mental lexicon, and also puts into perspective a number of assumptions underlying recently proposed distributional text-based models. In particular, we focus on word-embedding models which have been proposed to learn aspects of word meaning in a manner similar to humans. These are contrasted with internal language models derived from a new extensive data set of word associations. Using relatedness and similarity judgments we evaluate these models and find that the word-association-based internal language models consis- tently outperform current state-of-the art text-based external language models, often with a large margin. These results are not just a performance improvement; they also have implications for our understanding of how distributional knowledge is used by people. 1 Introduction How is semantic information encoded? How is similarity represented in the brain? And how can we capture this information computationally? One answer to this question involves distributional lexico- semantic models, which quantify the semantic similarity between lexical items based on their distri- butional properties in large samples of data. Recent models like word2vec (Mikolov et al., 2013) and GloVe (Pennington et al., 2014), which rely on external corpora as the source of data, increasingly appear to capture word meaning in ways that ever-more-closely resemble human representations. For instance, these models show systematic improvements over previous work in key benchmarks such as human sim- ilarity judgments of word pairs (Baroni et al., 2014). The strong performance of these models has also suggested to cognitive scientists that the learning mechanisms they embody might resemble how humans learn the meaning of some words (Mandera et al., in press). In this study we show that using word-association data instead of corpus data improves performance substantially above the current state-of-the-art. We suggest that this is because data-intensive distribu- tional models like word2vec, formidable though they are, may not capture word representations the way the average adult language speaker does. Their enormous, high-quality input data enables them to mimic human behavior, but they do relatively poorly compared to performance based on data that more accurately captures people’s true representations of meaning. The distinction between using text corpora or word association data maps onto the distinction made by Taylor (2012) between External language models (E-language) and Internal language models (I- language). An E-language model, like word2vec, treats language as …

Content truncated to 500 words