SimVerb-3500: A Large-Scale Evaluation Set of Verb Similarity

Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 2173–2182, Austin, Texas, November 1-5, 2016. c⃝2016 Association for Computational Linguistics SimVerb-3500: A Large-Scale Evaluation Set of Verb Similarity Daniela Gerz1, Ivan Vuli´c1, Felix Hill1, Roi Reichart2, and Anna Korhonen1 1Language Technology Lab, DTAL, University of Cambridge 2Faculty of Industrial Engineering and Management, Technion, IIT 1{dsg40,iv250,fh295,alk23}@cam.ac.uk 2roiri@ie.technion.ac.il Abstract Verbs play a critical role in the meaning of sentences, but these ubiquitous words have re- ceived little attention in recent distributional se- mantics research. We introduce SimVerb-3500, an evaluation resource that provides human ratings for the similarity of 3,500 verb pairs. SimVerb-3500 covers all normed verb types from the USF free-association database, pro- viding at least three examples for every Verb- Net class. This broad coverage facilitates de- tailed analyses of how syntactic and seman- tic phenomena together influence human un- derstanding of verb meaning. Further, with significantly larger development and test sets than existing benchmarks, SimVerb-3500 en- ables more robust evaluation of representation learning architectures and promotes the devel- opment of methods tailored to verbs. We hope that SimVerb-3500 will enable a richer under- standing of the diversity and complexity of verb semantics and guide the development of systems that can effectively represent and in- terpret this meaning. 1 Introduction Verbs are famously both complex and variable. They express the semantics of an event as well the rela- tional information among participants in that event, and they display a rich range of syntactic and seman- tic behaviour (Jackendoff, 1972; Gruber, 1976; Levin, 1993). Verbs play a key role at almost every level of linguistic analysis. Information related to their predi- cate argument structure can benefit many NLP tasks (e.g. parsing, semantic role labelling, information ex- traction) and applications (e.g. machine translation, text mining) as well as research on human language acquisition and processing (Korhonen, 2010). Precise methods for representing and understanding verb se- mantics will undoubtedly be necessary for machines to interpret the meaning of sentences with similar accuracy to humans. Numerous algorithms for acquiring word represen- tations from text and/or more structured knowledge bases have been developed in recent years (Mikolov et al., 2013; Pennington et al., 2014; Faruqui et al., 2015). These representations (or embeddings) typ- ically contain powerful features that are applicable to many language applications (Collobert and We- ston, 2008; Turian et al., 2010). Nevertheless, the predominant approaches to distributed representation learning apply a single learning algorithm and repre- sentational form for all words in a vocabulary. This is despite evidence that applying different learning algorithms to word types such as nouns, adjectives and verbs can significantly increase the ultimate use- fulness of representations (Schwartz et al., 2015). One factor behind the lack of more nuanced word representation learning methods is the scarcity of sat- isfactory ways to evaluate or analyse representations of particular word types. Resources such as MEN (Bruni et al., 2014), Rare Words (Luong et al., 2013) and SimLex-999 (Hill et al., 2015) focus either on words from a single …

Content truncated to 500 words