TyDi QA: A Benchmark for Information-Seeking Question Answering in Typologically Diverse Languages

TYDI QA: A Benchmark for Information-Seeking Question Answering in Typologically Diverse Languages Jonathan H. Clark␆␅ Eunsol Choi␅ Michael Collins␆ Dan Garrette␆ Tom Kwiatkowski␆ Vitaly Nikolaev␄␃ Jennimaria Palomaki␄␃ Google Research tydiqa@google.com Abstract Confidently making progress on multilingual modeling requires challenging, trustworthy evaluations. We present TYDI QA—a question answering dataset covering 11 typologically diverse languages with 204K question-answer pairs. The languages of TYDI QA are diverse with regard to their typology—the set of linguistic features each language expresses—- such that we expect models performing well on this set to generalize across a large num- ber of the world’s languages. We present a quantitative analysis of the data quality and example-level qualitative linguistic analyses of observed language phenomena that would not be found in English-only corpora. To pro- vide a realistic information-seeking task and avoid priming effects, questions are written by people who want to know the answer, but don’t know the answer yet, and the data is collected directly in each language without the use of translation. 1 Introduction When faced with a genuine information need, everyday users now benefit from the help of automatic question answering (QA) systems on a daily basis with high-quality systems integrated into search engines and digital assistants. Their questions are information-seeking—they want to know the answer, but don’t know the answer yet. Recognizing the need to align research with the impact it will have on real users, the community has responded with datasets of information- seeking questions such as WikiQA (Yang et al., 2015), MS MARCO (Nguyen et al., 2016), QuAC Pronounced tie dye Q. A.—like the colorful t-shirt. ␆Projectdesign ␅Modeling ␄Linguisticanalysis ␃Data quality. (Choi et al., 2018), and the Natural Questions (NQ) (Kwiatkowski et al., 2019). However, many people who might benefit from QA systems do not speak English. The lan- guages of the world exhibit an astonishing breadth of linguistic phenomena used to express mean- ing; the World Atlas of Language Structures (Comrie and Gil, 2005; Dryer and Haspelmath, 2013) categorizes over 2,600 languages1 by 192 typological features including phenomena such as word order, reduplication, grammatical mean- ings encoded in morphosyntax, case markings, plurality systems, question marking, relativiza- tion, and many more. If our goal is to build models that can accurately represent all human languages, we must evaluate these models on data that exemplifies this variety. In addition to these typological distinctions, modeling challenges arise due to differences in the availability of monolingual data, the availability of (expensive) parallel translation data, how standardized the writing system is variable spacing conventions (e.g., Thai), and more. With these needs in mind, we present the first public large- scale multilingual corpus of information-seeking question-answer pairs—using a simple-yet-novel data collection procedure that is model-free and translation-free. Our goals in doing so are: 1. to enable research progress toward building high-quality question answering systems in roughly the world’s top 100 languages;2 and 2. to encourage research on models that behave well across the linguistic phenomena and data scenarios of the world’s languages. We describe the typological features of …

Content truncated to 500 words