CogniVal: A Framework for Cognitive Word Embedding Evaluation

Proceedings of the 23rd Conference on Computational Natural Language Learning, pages 538–549 Hong Kong, China, November 3-4, 2019. c⃝2019 Association for Computational Linguistics 538 CogniVal: A Framework for Cognitive Word Embedding Evaluation Nora Hollenstein1, Antonio de la Torre1, Nicolas Langer2, Ce Zhang1 1 Department of Computer Science, ETH Zurich {noraho,antonide,ce.zhang}@ethz.ch 2 Department of Psychology, University of Zurich n.langer@psychologie.uzh.ch Abstract An interesting method of evaluating word rep- resentations is by how much they reflect the semantic representations in the human brain. However, most, if not all, previous works only focus on small datasets and a single modal- ity. In this paper, we present the first multi- modal framework for evaluating English word representations based on cognitive lexical se- mantics. Six types of word embeddings are evaluated by fitting them to 15 datasets of eye- tracking, EEG and fMRI signals recorded dur- ing language processing. To achieve a global score over all evaluation hypotheses, we ap- ply statistical significance testing accounting for the multiple comparisons problem. This framework is easily extensible and available to include other intrinsic and extrinsic evalu- ation methods. We find strong correlations in the results between cognitive datasets, across recording modalities and to their performance on extrinsic NLP tasks. 1 Introduction Word embeddings are the corner stones of state- of-the-art NLP models. Distributional represen- tations which interpret words, phrases, and sen- tences as high-dimensional vectors in semantic space have become increasingly popular. These vectors are obtained by training language models on large corpora to encode contextual information. Each vector represents the meaning of a word. Evaluating and comparing the quality of dif- ferent word embeddings is a well-known, largely open challenge. Currently, word embeddings are evaluated with extrinsic or intrinsic methods. Ex- trinsic evaluation is the process of assessing the quality of the embeddings based on their perfor- mance on downstream NLP tasks, such as ques- tion answering or entity recognition. However, embeddings can be trained and fine-tuned for spe- Figure 1: Overview of the cognitive word embedding evaluation process. cific tasks, but this does not mean that they accu- rately reflect the meaning of words. One the other hand, intrinsic evaluation meth- ods, such as word similarity and word analogy tasks, merely test single linguistic aspects. These tasks are based on conscious human judgements. Conscious judgements can be biased by subjec- tive factors and the tasks themselves might also be biased (Malvina Nissim, 2019). Additionally, the correlation between intrinsic and extrinsic metrics is not very clear, as intrinsic evaluation results fail 539 to predict extrinsic performance (Chiu et al., 2016; Gladkova and Drozd, 2016). Finally, both intrin- sic and extrinsic evaluation types often lack sta- tistical significance testing and do not provide a global quality score. In this paper, we focus on the intrinsic sub- conscious evaluation method (Bakarov, 2018b), which evaluates English word embeddings against the lexical representations of words in the hu- man brain, recorded when passively understand- ing language. Cognitive lexical semantics pro- poses that words are defined by how they are orga- nized …

Content truncated to 500 words