Bilingual alignment transfers to multilingual alignment for unsupervised parallel text mining

Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics Volume 1: Long Papers, pages 8696 - 8706 May 22-27, 2022 c⃝2022 Association for Computational Linguistics Bilingual alignment transfers to multilingual alignment for unsupervised parallel text mining Chih-chan Tien ∗ University of Chicago cctien@uchicago.edu Shane Steinert-Threlkeld University of Washington shanest@uw.edu Abstract This work presents methods for learning cross- lingual sentence representations using paired or unpaired bilingual texts. We hypothesize that the cross-lingual alignment strategy is transferable, and therefore a model trained to align only two languages can encode multilin- gually more aligned representations. We thus introduce dual-pivot transfer: training on one language pair and evaluating on other pairs. To study this theory, we design unsupervised models trained on unpaired sentences and single-pair supervised models trained on bi- texts, both based on the unsupervised language model XLM-R with its parameters frozen. The experiments evaluate the models as univer- sal sentence encoders on the task of unsuper- vised bitext mining on two datasets, where the unsupervised model reaches the state of the art of unsupervised retrieval, and the alterna- tive single-pair supervised model approaches the performance of multilingually supervised models. The results suggest that bilingual training techniques as proposed can be applied to get sentence representations with multilin- gual alignment. 1 Introduction Cross-lingual alignment as evaluated by retrieval tasks has been shown to be present in the repre- sentations of recent massive multilingual models which are not trained on bitexts (Pires et al., 2019; Conneau et al., 2020b). Other studies further show that sentence representations with higher cross- lingual comparability can be achieved by training a cross-lingual mapping (Aldarmaki and Diab, 2019) or fine-tuning (Cao et al., 2020) for every pair of languages. These two lines of research show that, on the one hand, multilingual alignment arises from training using monolingual corpora alone, and, on the other, bilingual alignment can be enhanced by training on bitexts of specific language pairs. ∗Work done while at the Department of Linguistics at the University of Washington. Combining these insights yields a question: can training with bilingual corpora help improve multi- lingual alignment? Given a language model encod- ing texts in different languages with some shared structure already, we can expect that the model fur- ther trained to align a pair of languages will take advantage of the shared structure and will there- fore generalize the alignment strategy to other lan- guage pairs. From a practical point of view, bitexts for some pairs of languages are more abundant than others, and it is therefore efficient to lever- age data from resource-rich pairs for the alignment of resource-poor pairs in training multilingual lan- guage models. To better understand the cross-lingual structure from the unsupervised models, we also ask the fol- lowing question: how can multilingual alignment information be extracted from the unsupervised language models? Unsupervised multilingual mod- els out-of-the-box as sentence encoders fall short of their supervised counterparts such as LASER (Artetxe and Schwenk, 2019b) in the task of bi- text mining (Hu et al., …

Content truncated to 500 words