A Resource-Free Evaluation Metric for Cross-Lingual Word Embeddings Based on Graph Modularity

Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 4952–4962 Florence, Italy, July 28 - August 2, 2019. c⃝2019 Association for Computational Linguistics 4952 A Resource-Free Evaluation Metric for Cross-Lingual Word Embeddings Based on Graph Modularity Yoshinari Fujinuma Computer Science University of Colorado fujinumay@gmail.com Jordan Boyd-Graber CS, iSchool, UMIACS, LSC University of Maryland jbg@umiacs.umd.edu Michael J. Paul Information Science University of Colorado mpaul@colorado.edu Abstract Cross-lingual word embeddings encode the meaning of words from different languages into a shared low-dimensional space. An important requirement for many downstream tasks is that word similarity should be indepen- dent of language—i.e., word vectors within one language should not be more similar to each other than to words in another language. We measure this characteristic using modular- ity, a network measurement that measures the strength of clusters in a graph. Modularity has a moderate to strong correlation with three downstream tasks, even though modularity is based only on the structure of embeddings and does not require any external resources. We show through experiments that modularity can serve as an intrinsic validation metric to im- prove unsupervised cross-lingual word embed- dings, particularly on distant language pairs in low-resource settings.1 1 Introduction The success of monolingual word embeddings in natural language processing (Mikolov et al., 2013b) has motivated extensions to cross-lingual settings. Cross-lingual word embeddings—where multiple languages share a single distributed representation—work well for classification (Kle- mentiev et al., 2012; Ammar et al., 2016) and ma- chine translation (Lample et al., 2018; Artetxe et al., 2018b), even with few bilingual pairs (Artetxe et al., 2017) or no supervision at all (Zhang et al., 2017; Conneau et al., 2018; Artetxe et al., 2018a). Typically the quality of cross-lingual word em- beddings is measured with respect to how well they improve a downstream task. However, sometimes it is not possible to evaluate embeddings for a spe- cific downstream task, for example a future task 1Our code is at https://github.com/akkikiki/ modularity_metric that does not yet have data or on a rare language that does not have resources to support traditional evaluation. In such settings, it is useful to have an intrinsic evaluation metric: a metric that looks at the embedding space itself to know whether the embedding is good without resorting to an extrinsic task. While extrinsic tasks are the ultimate arbiter of whether cross-lingual word embeddings work, intrinsic metrics are useful for low-resource lan- guages where one often lacks the annotated data that would make an extrinsic evaluation possible. However, few intrinsic measures exist for cross- lingual word embeddings, and those that do exist require external linguistic resources (e.g., sense- aligned corpora in Ammar et al. (2016)). The re- quirement of language resources makes this ap- proach limited or impossible for low-resource lan- guages, which are the languages where intrinsic evaluations are most needed. Moreover, requiring language resources can bias the evaluation toward words in the resources rather than evaluating the embedding space as a whole. Our solution involves a graph-based metric that considers the …

Content truncated to 500 words