Why Overfitting Isn’t Always Bad: Retrofitting Cross-Lingual Word Embeddings to Dictionaries

Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 2214–2220 July 5 - 10, 2020. c⃝2020 Association for Computational Linguistics 2214 Why Overfitting Isn’t Always Bad: Retrofitting Cross-Lingual Word Embeddings to Dictionaries Mozhi Zhang∗ CS and UMIACS University of Maryland mozhi@cs.umd.edu Yoshinari Fujinuma∗ Computer Science University of Colorado fujinumay@gmail.com Michael J. Paul Information Science University of Colorado mpaul@colorado.edu Jordan Boyd-Graber UMD CS, iSchool, UMIACS, and LSC and Google Research Zürich jbg@boydgraber.org Abstract Cross-lingual word embeddings (CLWE) are often evaluated on bilingual lexicon induc- tion (BLI). Recent CLWE methods use linear projections, which underfit the training dic- tionary, to generalize on BLI. However, un- derfitting can hinder generalization to other downstream tasks that rely on words from the training dictionary. We address this limitation by retrofitting CLWE to the training dictionary, which pulls training translation pairs closer in the embedding space and overfits the train- ing dictionary. This simple post-processing step often improves accuracy on two down- stream tasks, despite lowering BLI test accu- racy. We also retrofit to both the training dictio- nary and a synthetic dictionary induced from CLWE, which sometimes generalizes even bet- ter on downstream tasks. Our results confirm the importance of fully exploiting the training dictionary in downstream tasks and explains why BLI is a flawed CLWE evaluation. 1 Introduction Cross-lingual word embeddings (CLWE) map words across languages to a shared vector space. Recent supervised CLWE methods follow a projection-based pipeline (Mikolov et al., 2013). Using a training dictionary, a linear projection maps pre-trained monolingual embeddings to a multilingual space. While CLWE enable many mul- tilingual tasks (Klementiev et al., 2012; Guo et al., 2015; Zhang et al., 2016; Ni et al., 2017), most recent work only evaluates CLWE on bilingual lexi- con induction (BLI). Specifically, a set of test words are translated with a retrieval heuristic (e.g., near- est neighbor search) and compared against gold translations. BLI accuracy is easy to compute and captures the desired property of CLWE that transla- tion pairs should be close. However, BLI accuracy ∗⋆Equal contribution Retrofit Training Dictionary Original CLWE Updated CLWE Source Embedding Target Embedding Project Synthetic Dictionary Figure 1: To fully exploit the training dictionary, we retrofit projection-based CLWE to the training dictio- nary as a post-processing step (pink parts). To preserve correctly aligned translations in the original CLWE, we optionally retrofit to a synthetic dictionary induced from the original CLWE (orange parts). does not always correlate with accuracy on down- stream tasks such as cross-lingual document clas- sification and dependency parsing (Ammar et al., 2016; Fujinuma et al., 2019; Glavas et al., 2019). Let’s think about why that might be. BLI accu- racy is only computed on test words. Consequently, BLI hides linear projection’s inability to align all training translation pairs at once; i.e., projection- based CLWE underfit the training dictionary. Un- derfitting does not hurt BLI test accuracy, because test words are excluded from the training dictio- nary in BLI benchmarks. However, words from the training dictionary may be nonetheless predictive …

Content truncated to 500 words