MLQA: Evaluating Cross-lingual Extractive Question Answering

Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 7315–7330 July 5 - 10, 2020. c⃝2020 Association for Computational Linguistics 7315 MLQA: Evaluating Cross-lingual Extractive Question Answering Patrick Lewis*† Barlas O˘guz* Ruty Rinott* Sebastian Riedel*† Holger Schwenk* *Facebook AI Research †University College London {plewis,barlaso,ruty,sriedel,schwenk}@fb.com Abstract Question answering (QA) models have shown rapid progress enabled by the availability of large, high-quality benchmark datasets. Such annotated datasets are difficult and costly to collect, and rarely exist in languages other than English, making building QA systems that work well in other languages challeng- ing. In order to develop such systems, it is crucial to invest in high quality multilingual evaluation benchmarks to measure progress. We present MLQA, a multi-way aligned ex- tractive QA evaluation benchmark intended to spur research in this area.1 MLQA contains QA instances in 7 languages, English, Ara- bic, German, Spanish, Hindi, Vietnamese and Simplified Chinese. MLQA has over 12K in- stances in English and 5K in each other lan- guage, with each instance parallel between 4 languages on average. We evaluate state- of-the-art cross-lingual models and machine- translation-based baselines on MLQA. In all cases, transfer results are significantly behind training-language performance. 1 Introduction Question answering (QA) is a central and highly popular area in NLP, with an abundance of datasets available to tackle the problem from various angles, including extractive QA, cloze-completion, and open-domain QA (Richardson, 2013; Rajpurkar et al., 2016; Chen et al., 2017; Kwiatkowski et al., 2019). The field has made rapid advances in recent years, even exceeding human performance in some settings (Devlin et al., 2019; Alberti et al., 2019). Despite such popularity, QA datasets in lan- guages other than English remain scarce, even for relatively high-resource languages (Asai et al., 2018), as collecting such datasets at sufficient scale and quality is difficult and costly. There 1MLQA is publicly available at https://github. com/facebookresearch/mlqa are two reasons why this lack of data prevents in- ternationalization of QA systems. First, we can- not measure progress on multilingual QA with- out relevant benchmark data. Second, we cannot easily train end-to-end QA models on the task, and arguably most recent successes in QA have been in fully supervised settings. Given recent progress in cross-lingual tasks such as document classification (Lewis et al., 2004; Klementiev et al., 2012; Schwenk and Li, 2018), semantic role la- belling (Akbik et al., 2015) and NLI (Conneau et al., 2018), we argue that while multilingual QA training data might be useful but not strictly neces- sary, multilingual evaluation data is a must-have. Recognising this need, several cross-lingual datasets have recently been assembled (Asai et al., 2018; Liu et al., 2019a). However, these gen- erally cover only a small number of languages, combine data from different authors and annota- tion protocols, lack parallel instances, or explore less practically-useful QA domains or tasks (see Section 3). Highly parallel data is particularly attractive, as it enables fairer comparison across languages, requires fewer source language annota- tions, and allows for additional evaluation setups at no …

Content truncated to 500 words