FAST: A carefully sampled and cognitively motivated dataset for distributional semantic evaluation

Proceedings of the 25th Conference on Computational Natural Language Learning (CoNLL), pages 588–595 November 10–11, 2021. ©2021 Association for Computational Linguistics 588 FAST: A carefully sampled and cognitively motivated dataset for distributional semantic evaluation Stefan Evert Computational Linguistics Group FAU Erlangen-Nürnberg stefan.evert@fau.de Gabriella Lapesa Institute for Natural Language Processing University of Stuttgart lapesaga@ims.uni-stuttgart.de Abstract What is the first word that comes to your mind when you hear giraffe, or damsel, or freedom? Such free associations contain a huge amount of information on the mental representations of the corresponding concepts, and are thus an extremely valuable testbed for the evaluation of semantic representations extracted from cor- pora. In this paper, we present FAST (Free AS- sociation Tasks), a free association dataset for English rigorously sampled from two standard free association norms collections (the Edin- burgh Associative Thesaurus and the University of South Florida Free Association Norms), dis- cuss two evaluation tasks, and provide baseline results. In parallel, we discuss methodological considerations concerning the desiderata for a proper evaluation of semantic representations. 1 Introduction Assessing the performance of a distributional se- mantic model (DSM), be it a count model or one of the numerous and popular neural embeddings, has never been a straightforward endeavour. It is becoming an increasingly pressing issue due to the ‘black box’ nature of word embeddings and their aleatoric, often irreproducible training. We witness a proliferation of tasks and a growing dissatisfac- tion within the community: what are we modeling, really? Are models learning to handle a specific dataset, or a task, or do they really abstract seman- tic knowledge from text? This has lead to a shift towards evaluation methods capable of capturing human-like generalization (Linzen, 2020). DSM evaluation with standard datasets is af- fected by two major problems. The first issue is the possibility that DSMs may just exploit contingent properties of a task. For example, they could be learning to identify random controls instead of cap- turing the intended semantic relation (Evert, 2016). Even with carefully chosen distractors, models may just be picking up contingent features of the exper- imental items: Levy et al. (2015b) observe that many hypernym classifiers merely learn to recog- nize words that are ‘typical’ hypernyms, indepen- dently of the stimulus items (i.e. corresponding hyponyms). The second issue is dataset size, since small test sets inevitably lead to overfitting if many different model architectures or parameter settings are evaluated. Take the TOEFL synonym task, where a single item is worth 0.8 accuracy points, as an extreme example: Bullinaria and Levy (2012) achieve 100% accuracy after testing a wide range of DSM parameters. It is also true, however, that large datasets often lack in design quality (e.g. by using random controls, cf. above). In this paper, we present Free ASsociation Tasks (FAST), a new evaluation dataset for English lex- ical semantics designed to satisfy the desiderata highlighted above: (a) compatibility with human- like generalization, (b) quality of distractors, and (c) dataset size. We address (a) by focussing on a cognitive modeling task, …

Content truncated to 500 words