MKQA: A Linguistically Diverse Benchmark for Multilingual Open Domain Question Answering

MKQA: A Linguistically Diverse Benchmark for Multilingual Open Domain Question Answering Shayne Longpre Apple Inc. slongpre@mit.edu Yi Lu Apple Inc. ylu7@apple.com Joachim Daiber Apple Inc. jodaiber@apple.com Abstract Progress in cross-lingual modeling depends on challenging, realistic, and diverse evaluation sets. We introduce Multilingual Knowledge Questions and Answers (MKQA), an open- domain question answering evaluation set comprising 10k question-answer pairs aligned across 26 typologically diverse languages (260k question-answer pairs in total). An- swers are based on heavily curated, language- independent data representation, making results comparable across languages and independent of language-specific passages. With 26 lan- guages, this dataset supplies the widest range of languages to-date for evaluating question answering. We benchmark a variety of state- of-the-art methods and baselines for generative and extractive question answering, trained on Natural Questions, in zero shot and translation settings. Results indicate this dataset challenging even in English, but especially inis low-resource languages.1 1 Introduction Training and evaluation data for question answer- ing (QA) is severely lacking outside of high- resource languages like English. As unsupervised, transfer learning and zero/few-shot methods nar- row the multilingual performance gap with En- glish (Conneau et al., 2020; Lee and Lee, 2019; Cui et al., 2019a; Lewis et al., 2020), their real progress is hard to measure without challenging, realistic, and linguistically diverse evaluation sets. Existing multilingual QA datasets are realistic and challenging, but they lack linguistic diversity, comparable evaluation between languages, and are often limited to passages provided with the dataset (see Table 2). We introduce Multilingual Knowledge Ques- tions and Answers (MKQA) for evaluation of 1MKQA data and evaluation scripts are available at https://github.com/apple/ml-mkqa. open-domain question answering. MKQA selects 10k realistic English queries from the Natural Questions dataset (NQ, Kwiatkowski et al., 2019) and human translates them into 25 additional lan- guages and dialects. Accompanying these query translations we replace NQ’s passage embedded answer spans with high-quality, language- and retrieval-independent answer annotations, linked directly against Wikidata entities and a limited set of well-defined value types (numbers, dates, strings, etc.).2 See one full example in Table 1. More flexi- ble than existing multilingual datasets, MKQA’s grading procedure ensures these labels are suf- ficient to evaluate any QA method, including knowledge graph and generative approaches. The objective of this evaluation set is to facilitate fair comparison between languages, without imposing assumptions on the underlying QA approach. We see MKQA as a useful tool enabling practition- ers to benchmark a variety of multilingual open domain question answering methods against the widest range of available languages yet. Below, we discuss its central properties as an evaluation benchmark. Realistic and Reliable Annotations Of crucial importance to any evaluation set is (a) how well it reflects realistic, real-world settings, and (b) the reliability of its annotations. To ensure the English queries, which form the basis of our dataset, are realistic, we use Natural Questions, formulated by real users, independent of passages or answers. To ensure these queries are realistic in other languages we employ expert bilingual translators, guided by strict localization criteria. We confirm that …

Content truncated to 500 words