What just happened? Evaluating retrofitted distributional word vectors

Proceedings of NAACL-HLT 2019, pages 1062–1072 Minneapolis, Minnesota, June 2 - June 7, 2019. c⃝2019 Association for Computational Linguistics 1062 What just happened? Evaluating retrofitted distributional word vectors Dmetri Hayes International Computer Science Institute University of California, Berkeley Berkeley, CA, 94720, USA dmetri@berkeley.edu Abstract Recent work has attempted to enhance vector space representations using information from structured semantic resources. This process, dubbed retrofitting Faruqui et al. (2015), has yielded improvements in word similarity per- formance. Research has largely focused on the retrofitting algorithm, or on the kind of struc- tured semantic resources used, but little re- search has explored why some resources per- form better than others. We conducted a fine- grained analysis of the original retrofitting pro- cess, and found that the utility of different lex- ical resources for retrofitting depends on two factors: the coverage of the resource and the evaluation metric. Our assessment suggests that the common practice of using correla- tion measures to evaluate increases in perfor- mance against full word similarity benchmarks 1) obscures the benefits offered by smaller re- sources, and 2) overlooks incremental gains in word similarity performance. We propose root-mean-square error (RMSE) as an alterna- tive evaluation metric, and demonstrate that correlation measures and RMSE sometimes yield opposite conclusions concerning the effi- cacy of retrofitting. This point is illustrated by word vectors retrofitted with novel treatments of the FrameNet data (Fillmore and Baker, 2010). 1 Introduction One of the most challenging tasks in the field of Natural Language Processing (NLP) is accu- rately encoding meaning into a computational sys- tem. Currently, the predominant approach is to represent the meanings of linguistic units, such as words or phrases, as vectors in a high-dimensional space. Vector embeddings are trained over large text corpora using machine-learning techniques, and have proven useful for a wide range of appli- cations, such as named entity recognition (Turian et al., 2010), semantic role labeling (Collobert et al., 2011), sentiment analysis (Socher et al., 2013), and machine translation (Zou et al., 2013). Word vectors are typically trained solely on the distributional information from text corpora. Re- cent work has attempted to improve word vec- tors by infusing them with information from se- mantic resources in a post-processing step. This technique, referred to as retrofitting, was intro- duced by Faruqui et al. (2015). They adjusted pre- trained embeddings based on lexical relations in WordNet (Miller, 1995), FrameNet (Fillmore and Baker, 2010), and the Paraphrase Database (Gan- itkevitch et al., 2013). In some cases, this method yielded gains in word similarity performance. Retrofitting has been extended in a variety of ways. Briefly, these include 1) adding word-to- word relations to encompass more than just sim- ilarity relations, such as by directly introducing antonymy relations (Mrkˇsi´c et al., 2016), or by ex- plicitly modeling the pairwise relations between items (Lengerich et al., 2017); 2) increasing the size of the output vocabulary (Speer et al., 2017), or extending the process to affect the word vec- tors of words outside of the semantic resource (Glavaˇs …

Content truncated to 500 words