Evaluation by Association: A Systematic Study of Quantitative Word Association Evaluation

Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics: Volume 1, Long Papers, pages 163–175, Valencia, Spain, April 3-7, 2017. c⃝2017 Association for Computational Linguistics Evaluation by Association: A Systematic Study of Quantitative Word Association Evaluation Ivan Vuli´c1, Douwe Kiela2, and Anna Korhonen1 1 Language Technology Lab, DTAL, University of Cambridge 2 Facebook AI Research {iv250|alk23}@cam.ac.uk dkiela@fb.com Abstract Recent work on evaluating representation learning architectures in NLP has estab- lished a need for evaluation protocols based on subconscious cognitive measures rather than manually tailored intrinsic similarity and relatedness tasks. In this work, we pro- pose a novel evaluation framework that en- ables large-scale evaluation of such archi- tectures in the free word association (WA) task, which is firmly grounded in cognitive theories of human semantic representation. This evaluation is facilitated by the exis- tence of large manually constructed reposi- tories of word association data. In this pa- per, we (1) present a detailed analysis of the new quantitative WA evaluation protocol, (2) suggest new evaluation metrics for the WA task inspired by its direct analogy with information retrieval problems, (3) evaluate various state-of-the-art representation mod- els on this task, and (4) discuss the relation- ship between WA and prior evaluations of semantic representation with well-known similarity and relatedness evaluation sets. We have made the WA evaluation toolkit publicly available. 1 Introduction The quality of word representations in semantic models is often measured using intrinsic evalua- tions that capture particular types of relationships (typically semantic similarity and relatedness) be- tween word pairs (Finkelstein et al., 2002; Hill et al., 2015; Schnabel et al., 2015; Tsvetkov et al., 2015, inter alia). Whereas the notions of semantic similarity and relatedness constitute key concepts in such evalua- tions, they are in fact vaguely defined (Batchkarov et al., 2016; Ettinger and Linzen, 2016). The con- struction of ground truth evaluation sets that reflect these relations, such as SimLex-999 (Hill et al., 2015), SimVerb-3500 (Gerz et al., 2016), MEN (Bruni et al., 2014) or Rare Words (Luong et al., 2013), relies on manually constructed guidelines that trigger subjective human interpretation of the task at hand. This in turn introduces inter-annotator variability (Batchkarov et al., 2016) and does not account for the fact that human similarity judge- ments are asymmetric by nature (Tversky, 1977). What is more, given that humans perform lin- guistic comparisons between concepts on a sub- conscious level (Kutas and Federmeier, 2011), it is at least debatable whether current similar- ity/relatedness evaluation sets fully capture the im- plicit relational structure underlying human lan- guage representation and understanding. As evidenced by recent workshops on evalua- tion of semantic representations1, the community appears to recognise that current evaluation meth- ods are inadequate. To fill in this gap, recent work has proposed using subconscious cognitive mea- sures of semantic connection instead, as a proxy for measuring the ability of statistical models to tackle various problems in human language un- derstanding (Ettinger and Linzen, 2016; Søgaard, 2016; Mandera et al., 2017). Motivated …

Content truncated to 500 words