Evaluating Word Embeddings with Categorical Modularity

Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021, pages 1982–1993 August 1–6, 2021. ©2021 Association for Computational Linguistics 1982 Evaluating Word Embeddings with Categorical Modularity S´ılvia Casacuberta1∗, Karina Halevy1∗, Dami´an E. Blasi1,2,3 1: Harvard University 2: MPI for Evolutionary Anthropology 3: HSE University {scasacubertapuig, khalevy}@college.harvard.edu dblasi@fas.harvard.edu Abstract We introduce categorical modularity, a novel low-resource intrinsic metric to evaluate word embedding quality. Categorical modularity is a graph modularity metric based on the k-near- est neighbor graph constructed with embed- ding vectors of words from a fixed set of se- mantic categories, in which the goal is to mea- sure the proportion of words that have nearest neighbors within the same categories. We use a core set of 500 words belonging to 59 neuro- biologically motivated semantic categories in 29 languages and analyze three word embed- ding models per language (FastText, MUSE, and subs2vec). We find moderate to strong positive correlations between categorical mod- ularity and performance on the monolingual tasks of sentiment analysis and word similar- ity calculation and on the cross-lingual task of bilingual lexicon induction both to and from English. Overall, we suggest that cate- gorical modularity provides non-trivial predic- tive information about downstream task per- formance, with breakdowns of correlations by model suggesting some meta-predictive prop- erties about semantic information loss as well. 1 Introduction Word embeddings represent words and phrases in continuous low-dimensional vector spaces. They are usually trained with neural language models or word collocations (Bengio et al., 2003; Collobert and Weston, 2008; Mikolov et al., 2013b; Levy et al., 2015), such that similar assignments in a space reflect similar usage patterns. The rise of mono- lingual embeddings such as Word2Vec (Mikolov et al., 2013b), GloVe (Pennington et al., 2014), and FastText (Bojanowski et al., 2017), coupled with the need to transfer lexical knowledge across languages, has also led to the development of cross- lingual word embeddings, in which different lan- ∗Equal contribution guages share a single distributed representation and are mapped into the same vector space. Such meth- ods use different bilingual supervision signals (at the level of words, sentences, or documents) with varying levels of strength (Ruder et al., 2019). A central task in the study of word embeddings is finding metrics to evaluate their quality. These metrics can either be extrinsic, where embeddings are used as input features for downstream NLP tasks and evaluated on their performance, or in- trinsic, where embeddings are directly tested for how well they capture syntactic or semantic proper- ties in their own right (Qiu et al., 2018). Extrinsic methods are not always feasible for low-resource languages due to a lack of annotated data. More- over, downstream model components can be fine- tuned to achieve higher performance on certain tasks without necessarily indicating improvement in the semantic representation of words in an em- bedding space (Leszczynski et al., 2020). This paper presents categorical modularity, a low-resource intrinsic evaluation metric for both monolingual and cross-lingual word embeddings based on the notion of graph modularity. The under- lying principle …

Content truncated to 500 words