Knowledge Distillation for Bilingual Dictionary Induction

Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 2497–2506 Copenhagen, Denmark, September 7–11, 2017. c⃝2017 Association for Computational Linguistics Knowledge Distillation for Bilingual Dictionary Induction Ndapandula Nakashole and Raphael Flauger Computer Science and Engineering University of California, San Diego La Jolla, CA 92093 nnakashole@eng.ucsd.edu Abstract Leveraging zero-shot learning to learn mapping functions between vector spaces of different languages is a promising ap- proach to bilingual dictionary induction. However, methods using this approach have not yet achieved high accuracy on the task. In this paper, we propose a bridg- ing approach, where our main contribution is a knowledge distillation training objec- tive. As teachers, rich resource transla- tion paths are exploited in this role. And as learners, translation paths involving low resource languages learn from the teach- ers. Our training objective allows seam- less addition of teacher translation paths for any given low resource pair. Since our approach relies on the quality of monolin- gual word embeddings, we also propose to enhance vector representations of both the source and target language with linguistic information. Our experiments on various languages show large performance gains from our distillation training objective, ob- taining as high as 17% accuracy improve- ments. 1 Introduction In traditional supervised learning, a classifier is trained on a labeled dataset of the form (X, Y). Each xi ∈ X is a feature vector representing a single training instance and yi ∈ Y is the label as- sociated with xi. In zero-shot learning (Mitchell et al., 2008), at test time we can encounter a test instance xj whose corresponding label was not seen at training time. This setting occurs in do- mains where Y can take on many values, and ob- taining labeled examples for all possible Y values is expensive. Computer vision is one such do- main, where there are thousands of objects a sys- tem needs to recognize yet at training time we may only see examples of some of the objects. In zero- shot learning, instead of learning parameters as- sociated with each possible label in Y , the learn- ing task is cast as a problem of learning a single mapping function from the vector space of input instances to the vector space of the output labels. The resulting induced function can then be applied to test instances xj whose labels may not have been seen at training time, producing a projected vector, ˆyj, in the label space. The nearest neigh- bor of the mapped vector in the label space is then considered to be the label of xj. In this paper, we study zero-shot learning in the context of bilingual dictionary induction, which is the problem of mapping words from a source language to equivalent words in a target language. The label space is the full vocabulary of the target language which can be on the order of millions of tokens. First, word embeddings are learned separately for each language, and second, using a given seed dictionary, we train a mapping …

Content truncated to 500 words