Combining Static Word Embeddings and Contextual Representations for Bilingual Lexicon Induction

Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021, pages 2943–2955 August 1–6, 2021. ©2021 Association for Computational Linguistics 2943 Combining Static Word Embeddings and Contextual Representations for Bilingual Lexicon Induction Jinpeng Zhang1, Baijun Ji1, Nini Xiao1, Xiangyu Duan1∗, Min Zhang1 Yangbin Shi2, Weihua Luo2 1 Institute of Aritificial Intelligence, School of Computer Science and Technology, Soochow university 2 Alibaba DAMO Academy {jpzhang1,bjji,nnxiaoxiao}@stu.suda.edu.cn; {xiangyuduan,minzhang}@suda.edu.cn; taiwu.syb@taobao.com; weihua.luowh@alibaba-inc.com Abstract Bilingual Lexicon Induction (BLI) aims to map words in one language to their transla- tions in another, and are typically through learning linear projections to align monolin- gual word representation spaces. Two classes of word representations have been explored for BLI: static word embeddings and contextual representations, but there is no studies to com- bine both. In this paper, we propose a sim- ple yet effective mechanism to combine the static word embeddings and the contextual rep- resentations to utilize the advantages of both paradigms. We test the combination mecha- nism on various language pairs under the su- pervised and unsupervised BLI benchmark set- tings. Experiments show that our mechanism consistently improves performances over ro- bust BLI baselines on all language pairs by av- eragely improving 3.2 points in the supervised setting, and 3.1 points in the unsupervised set- ting1. 1 Introduction Bilingual Lexicon Induction (BLI) aims to find bilingual translation lexicons from monolingual corpora in two languages (Haghighi et al., 2008; Xing et al., 2015; Zhang et al., 2017a; Artetxe et al., 2017; Conneau et al., 2017), and is applied on nu- merous NLP tasks such as POS tagging (Zhang et al., 2016), parsing (Xiao and Guo, 2014), and ma- chine translation (Irvine and Callison-Burch, 2013; Qi et al., 2018). Most work on BLI learns a mapping between two static word embedding spaces, which are pre- trained on large monolingual corpora (Ruder et al., 2019). Both linear mapping (Mikolov et al., 2013; Xing et al., 2015; Artetxe et al., 2016; Smith et al., 2017) and non-linear mapping (Mohiuddin et al., ∗ Corresponding Author. 1Code is released at https://github.com/zjpbinary/CSCBLI 2020) methods have been studied to align the two spaces. Recently, other than the static word embed- dings, contextual representations are used for BLI due to the significant progresses on cross-lingual applications (Aldarmaki and Diab, 2019; Schuster et al., 2019). Although the static word embed- dings and the contextual representations exhibit properties suited for alignment, there is no works to combine the two paradigms. On one hand, the static word embeddings have been widely used for BLI, but one specific embed- ding mapping function does not ensure that in all conditions, words in a translation pair are nearest neighbors in the mapped common space. On the other hand, the contextual representations contain rich semantic information beneficial for alignment, but the dynamic contexts of word tokens pose a challenge for aligning word types. In this paper, we propose a combination mech- anism to utilize the static word embeddings and the contextual representations simultaneously. The combination mechanism consists of two parts. The first part is the …

Content truncated to 500 words