Learning Translation Models from Monolingual Continuous Representations

Human Language Technologies: The 2015 Annual Conference of the North American Chapter of the ACL, pages 1527–1536, Denver, Colorado, May 31 – June 5, 2015. c⃝2015 Association for Computational Linguistics Learning Translation Models from Monolingual Continuous Representations Kai Zhao∗ Graduate Center, CUNY New York, NY 10016, USA kzhao.hf@gmail.com Hany Hassan Microsoft Research Redmond, WA 98502, USA hanyh@microsoft.com Michael Auli∗ Facebook AI Research Menlo Park, CA 94025, USA michaelauli@fb.com Abstract Translation models often fail to generate good translations for infrequent words or phrases. Previous work attacked this problem by in- ducing new translation rules from monolin- gual data with a semi-supervised algorithm. However, this approach does not scale very well since it is very computationally expen- sive to generate new translation rules for only a few thousand sentences. We propose a much faster and simpler method that directly hallu- cinates translation rules for infrequent phrases based on phrases with similar continuous rep- resentations for which a translation is known. To speed up the retrieval of similar phrases, we investigate approximated nearest neighbor search with redundant bit vectors which we find to be three times faster and significantly more accurate than locality sensitive hashing. Our approach of learning new translation rules improves a phrase-based baseline by up to 1.6 BLEU on Arabic-English translation, it is three-orders of magnitudes faster than existing semi-supervised methods and 0.5 BLEU more accurate. 1 Introduction Statistical translation models (Koehn et al. 2003, Chiang et al. 2005) are trained with bilingual data and a simple solution to improve accuracy is to train on more data. However, for many language pairs we only have a very limited amount of bilingual data and even when dealing with resource-rich lan- guages, we still often perform poorly when dealing with rare words or phrases. On the other hand, there is plenty of monolingual data and previous work has investigated its use in learning translation models (Rapp, 1995; Callison- Burch et al., 2006; Haghighi et al., 2008; Saluja et *The entirety of this work was conducted while at Mi- crosoft Research. al., 2014). However, most methods rely on statistics that are computationally expensive. As a concrete example, the graph propagation algorithm of Saluja et al. (2014) relies on pair-wise mutual information statistics between any pair of phrases in the monolin- gual corpus that is very expensive to compute, even for moderately sized corpora. In this paper, we study the use of standard con- tinuous representations for words to generate trans- lation rules for infrequent phrases (§2). We ex- plore linear projections that map continuous repre- sentations of rare foreign phrases to English phrases. In particular, we propose to learn many local pro- jections that are specific to a given foreign phrase. We find this to be much more accurate than a sin- gle globally learned mapping such as proposed by (Mikolov et al. 2013; §3). Our method relies on the fast retrieval of simi- lar phrases in continuous space. We explore both Locality Sensitive Hashing (LSH; Indyk and Mot- wani, 2008) as well …

Content truncated to 500 words