Interactive Refinement of Cross-Lingual Word Embeddings

Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing, pages 5984–5996, November 16–20, 2020. c⃝2020 Association for Computational Linguistics 5984 Interactive Refinement of Cross-Lingual Word Embeddings Michelle Yuan∗ University of Maryland myuan@cs.umd.edu Mozhi Zhang∗ University of Maryland mozhi@cs.umd.edu Benjamin Van Durme Johns Hopkins University vandurme@jhu.edu Leah Findlater University of Washington leahkf@uw.edu Jordan Boyd-Graber University of Maryland jbg@umiacs.umd.edu Abstract Cross-lingual word embeddings transfer knowledge between languages: models trained on high-resource languages can pre- dict in low-resource languages. We introduce CLIME, an interactive system to quickly refine cross-lingual word embeddings for a given classification problem. First, CLIME ranks words by their salience to the downstream task. Then, users mark similarity between keywords and their nearest neighbors in the embedding space. Finally, CLIME updates the embeddings using the annotations. We evaluate CLIME on identifying health-related text in four low-resource languages: Ilocano, Sinhalese, Tigrinya, and Uyghur. Embeddings refined by CLIME capture more nuanced word semantics and have higher test accuracy than the original embeddings. CLIME often improves accuracy faster than an active learning baseline and can be easily combined with active learning to improve results. 1 Introduction Modern text classification requires large labeled datasets and pre-trained word embeddings (Kim, 2014; Iyyer et al., 2015; Joulin et al., 2017). How- ever, scarcity of both labeled and unlabeled data holds back applications in low-resource languages. Cross-lingual word embeddings (Mikolov et al., 2013a, CLWE) can bridge the gap by mapping words from different languages to a shared vec- tor space. Using CLWE features, models trained in a resource-rich language (e.g., English) can predict labels for other languages. The success of CLWE relies on the domain and quality of training data (Søgaard et al., 2018). While these methods have impressive word trans- lation accuracy, they are not tailored for down- stream tasks such as text classification (Glavas ∗⋆ indicates equal contribution mediocre lackluster disappointing uninspiring ����� ���� �������� ��������� ����������� ������ ������� ������� Figure 1: A hypothetical topographic map of an English–French embedding space tailored for senti- ment analysis. Dots are English words, and squares are French words. Positive sentiment words are grouped in a clime (red), while negative sentiment words are grouped in another clime (blue). These climes help sen- timent analysis. et al., 2019; Zhang et al., 2020a). We de- velop CLassifying Interactively with Multilingual Embeddings (CLIME), that efficiently specializes CLWE with human interaction.1 Given a pre-trained CLWE, a bilingual speaker in the loop reviews the nearest-neighbor words. CLIME capitalizes on the intuition that neighboring words in an ideal embed- ding space should have similar semantic attributes. In an analogy to geographic climes—zones with distinctive meteorological features—we call areas in the embedding space where words share similar semantic features climes. Our goal is to convert neighborhoods in the embedding space into classifi- cation climes with words that induce similar labels for a given classification task. For example, in the embedding for English–French sentiment analy- sis, positive sentiment words such as “excellent”, “exceptional”, and their French translations are to- gether, while “disappointing”, “lackluster”, and …

Content truncated to 500 words