Dual-Alignment Pre-training for Cross-lingual Sentence Embedding

Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics Volume 1: Long Papers, pages 3466–3478 July 9-14, 2023 ©2023 Association for Computational Linguistics Dual-Alignment Pre-training for Cross-lingual Sentence Embedding Ziheng Li1,∗, Shaohan Huang2, Zihan Zhang2, Zhi-Hong Deng1,†, Qiang Lou2, Haizhen Huang2, Jian Jiao2, Furu Wei2, Weiwei Deng2, Qi Zhang2 1School of Intelligence Science and Technology, Peking University, Beijing, China 2Microsoft Corporation {liziheng,zhdeng}@pku.edu.cn {shaohanh, zihzha, qilou, hhuang, jiajia, fuwei, dedeng, qizhang}@microsoft.com Abstract Recent studies have shown that dual encoder models trained with the sentence-level trans- lation ranking task are effective methods for cross-lingual sentence embedding. However, our research indicates that token-level align- ment is also crucial in multilingual scenar- ios, which has not been fully explored previ- ously. Based on our findings, we propose a dual-alignment pre-training (DAP) framework for cross-lingual sentence embedding that in- corporates both sentence-level and token-level alignment. To achieve this, we introduce a novel representation translation learning (RTL) task, where the model learns to use one-side contextualized token representation to recon- struct its translation counterpart. This recon- struction objective encourages the model to embed translation information into the token representation. Compared to other token-level alignment methods such as translation language modeling, RTL is more suitable for dual en- coder architectures and is computationally effi- cient. Extensive experiments on three sentence- level cross-lingual benchmarks demonstrate that our approach can significantly improve sentence embedding. Our code is available at https://github.com/ChillingDream/DAP. 1 Introduction Cross-lingual sentence embedding encodes mul- tilingual texts into a single unified vector space for a variety of Natural Language Processing (NLP) tasks, including cross-lingual sentence re- trieval (Artetxe and Schwenk, 2019b) and cross- lingual natural language inference (Conneau et al., 2018). The text sequences can be efficiently re- trieved and compared using the inner product be- tween their dense representations. The task of sentence embedding now heavily depends on pre-trained language models (Devlin ∗Work done during internship at Microsoft. †Corresponding Author. (a) Sentence Alignment. (b) Dual Alignment. Figure 1: Visualization of token representations of 100 Tatoeba sentence pairs from Arabic and English. The high-dimensional vectors are projected onto a 2D space by Principle Component Analysis. We show the results of two models fine-tuned from multilingual BERT. The model shown in Figure 1(a) only fine-tunes with the translation ranking task, resulting in large misaligned areas. This misalignment can be effectively eliminated by the proposed RTL methods as shown in 1(b). et al., 2019; Conneau and Lample, 2019; Conneau et al., 2020b,a). By fine-tuning the CLS token of the pre-trained model, they encode the input text sequence into a single vector representation. Recent research has shown that using the transla- tion ranking task in combination with a dual pre- trained encoder can result in superior sentence em- beddings (Yang et al., 2019; Chidambaram et al., 2019; Yang et al., 2021; Chi et al., 2021; Feng et al., 2022). The purpose of fine-tuning the CLS token is to learn sentence-level alignment and to compress the entire sentence’s information into the CLS token. This method makes the CLS …

Content truncated to 500 words