Lightweight Cross-Lingual Sentence Representation Learning

Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing, pages 2902–2913 August 1–6, 2021. ©2021 Association for Computational Linguistics 2902 Lightweight Cross-Lingual Sentence Representation Learning Zhuoyuan Mao♠ Prakhar Gupta♣ Chenhui Chu♠ Martin Jaggi♣ Sadao Kurohashi♠ ♠Kyoto University, Japan ♣EPFL, Switzerland {zhuoyuanmao, chu, kuro}@nlp.ist.i.kyoto-u.ac.jp {prakhar.gupta, martin.jaggi}@epfl.ch Abstract Large-scale models for learning fixed- dimensional cross-lingual sentence represen- tations like LASER (Artetxe and Schwenk, 2019b) lead to significant improvement in performance on downstream tasks. However, further increases and modifications based on such large-scale models are usually impracti- cal due to memory limitations. In this work, we introduce a lightweight dual-transformer architecture with just 2 layers for generating memory-efficient cross-lingual sentence repre- sentations. We explore different training tasks and observe that current cross-lingual training tasks leave a lot to be desired for this shallow architecture. To ameliorate this, we propose a novel cross-lingual language model, which combines the existing single-word masked language model with the newly proposed cross-lingual token-level reconstruction task. We further augment the training task by the introduction of two computationally-lite sentence-level contrastive learning tasks to en- hance the alignment of cross-lingual sentence representation space, which compensates for the learning bottleneck of the lightweight transformer for generative tasks. Our compar- isons with competing models on cross-lingual sentence retrieval and multilingual document classification confirm the effectiveness of the newly proposed training tasks for a shallow model. 1 1 Introduction Cross-lingual sentence representation mod- els (Schwenk and Douze, 2017; Espa˜na-Bonet et al., 2017; Yu et al., 2018; Devlin et al., 2019; Chidambaram et al., 2019; Artetxe and Schwenk, 2019b; Kim et al., 2019; Sabet et al., 2019; Conneau and Lample, 2019; Feng et al., 2020; Li 1https://github.com/Mao-KU/ lightweight-crosslingual-sent2vec and Mak, 2020) learn language-agnostic represen- tations facilitating tasks like cross-lingual sentence retrieval (XSR) and cross-lingual knowledge transfer on downstream tasks without the need for training a new monolingual representation model from scratch. Thus, such models benefit from an increased amount of data during training and lead to improved performances for low-resource languages. The above-mentioned models can be categorized into two classes. On one hand, global fine-tuning methods like mBERT (Devlin et al., 2019) and XLM (Conneau and Lample, 2019) require being fine-tuned globally which results in a significant overhead of its own. On the other hand, fixed- dimensional methods like LASER (Artetxe and Schwenk, 2019b) fix the sentence representations during the pre-training phase, and subsequently the fine-tuning for specific downstream tasks without back-propagating to the pre-trained model will be extremely computationally-lite. Lightweight mod- els have been sufficiently explored for the former group by either shrinking the model (Lan et al., 2020) or training a student model (Sanh et al., 2019; Jiao et al., 2020; Reimers and Gurevych, 2020; Sun et al., 2020). However, the lightweight models for the latter group have not been explored before, which may have a more promising future for de- ploying task-specific fine-tuning onto edge devices. In this work, we propose a variety of train- ing tasks for …

Content truncated to 500 words