LAReQA: Language-Agnostic Answer Retrieval from a Multilingual Pool

Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing, pages 5919–5930, November 16–20, 2020. c⃝2020 Association for Computational Linguistics 5919 LAReQA: Language-Agnostic Answer Retrieval from a Multilingual Pool Uma Roy‡ ∗ Noah Constant∗ Rami Al-Rfou Aditya Barua Aaron Phillips Yinfei Yang Google Research uma.roy.us@gmail.com {nconstant,rmyeid,adityabarua,aaronbphillips,yinfeiy}@google.com Abstract We present LAReQA, a challenging new benchmark for language-agnostic answer re- trieval from a multilingual candidate pool. Un- like previous cross-lingual tasks, LAReQA tests for “strong” cross-lingual alignment, re- quiring semantically related cross-language pairs to be closer in representation space than unrelated same-language pairs. This level of alignment is important for the practical task of cross-lingual information retrieval. Build- ing on multilingual BERT (mBERT), we study different strategies for achieving strong align- ment. We find that augmenting training data via machine translation is effective, and im- proves significantly over using mBERT out- of-the-box. Interestingly, model performance on zero-shot variants of our task that only target “weak” alignment is not predictive of performance on LAReQA. This finding un- derscores our claim that language-agnostic re- trieval is a substantively new kind of cross- lingual evaluation, and suggests that measur- ing both weak and strong alignment will be important for improving cross-lingual systems going forward. We release our dataset and evaluation code at https://github.com/ google-research-datasets/lareqa. 1 Introduction Recent progress in self-supervised pretraining for language understanding has enabled training large multilingual models on 100+ languages at the same time, as in multilingual BERT (mBERT) and XLM-R (Devlin et al., 2019; Conneau et al., 2019). These models, despite being trained without any ex- plicit objective of cross-lingual alignment, are sur- prisingly effective for cross-lingual transfer (Pires ‡Work done as a Google AI Resident. ∗Corresponding authors: uma.roy.us@gmail.com, nconstant@google.com Spanish English Legend Question Answer English Spanish Correct Answer Incorrect Answer (a) Weak Alignment (b) Strong Alignment Figure 1: A weakly aligned multilingual embedding space enables zero-shot transfer between languages, but incorrect answers in the same language are pre- ferred over correct answers in a different language. A strongly aligned embedding space “factors out” lan- guage, so the most semantically relevant pairs are al- ways the closest, regardless of language. et al., 2019; Wu and Dredze, 2019; Conneau et al., 2019), suggesting that the models may have learned to “factor out” language and embed inputs into a language-agnostic space. At the same time, Wu and Dredze (2019) observe that mBERT representations at all layers are highly accurate (>96%) at classifying language ID, which the authors find “surprising given the model’s zero- shot cross-lingual abilities”. This raises an inter- esting question. To what degree are models like mBERT and XLM-R language agnostic or easily adaptable to be so? Have they effectively disen- tangled the language-specific signal from the un- derlying semantic content, with each occupying a separate subspace? Can the learned representations be adapted by a lightweight alignment procedure to be truly language agnostic? To address these questions, we introduce a chal- lenging new task, LAReQA: Language Agnos- tic Retrieval Question Answering, which requires models to retrieve relevant cross-lingual answers …

Content truncated to 500 words