EXAMS: A Multi-subject High School Examinations Dataset for Cross-lingual and Multilingual Question Answering

Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing, pages 5427–5444, November 16–20, 2020. c⃝2020 Association for Computational Linguistics 5427 EXAMS: A Multi-Subject High School Examinations Dataset for Cross-Lingual and Multilingual Question Answering Momchil Hardalov1 Todor Mihaylov2 Dimitrina Zlatkova1 Yoan Dinkov1 Ivan Koychev1 Preslav Nakov3 1Sofia University “St. Kliment Ohridski”, Bulgaria, 2Heidelberg University, Germany 3Qatar Computing Research Institute, HBKU, Qatar, {hardalov, koychev}@fmi.uni-sofia.bg {dvzlatkova, jdinkov}@uni-sofia.bg tbmihaylov@gmail.com, pnakov@hbku.edu.qa Abstract We propose Eχαµs – a new benchmark dataset for cross-lingual and multilingual question answering for high school examina- tions. We collected more than 24,000 high- quality high school exam questions in 16 lan- guages, covering 8 language families and 24 school subjects from Natural Sciences and So- cial Sciences, among others. Eχαµs offers a fine-grained evaluation frame- work across multiple languages and subjects, which allows precise analysis and comparison of various models. We perform various ex- periments with existing top-performing multi- lingual pre-trained models and we show that Eχαµs offers multiple challenges that require multilingual knowledge and reasoning in mul- tiple domains. We hope that Eχαµs will en- able researchers to explore challenging reason- ing and knowledge transfer methods and pre- trained models for school question answer- ing in various languages which was not pos- sible before. The data, code, pre-trained mod- els, and evaluation are available at http:// github.com/mhardalov/exams-qa. 1 Introduction Research on science question answering has at- tracted a lot of attention in recent years (Clark, 2015; Schoenick et al., 2017; Clark et al., 2019). Such questions are challenging as they require do- main and common sense knowledge (Clark et al., 2018), as well as complex reasoning and differ- ent forms of inference over a variety of knowl- edge sources (Khashabi et al., 2016, 2018). In- deed, a combination of these was required to achieve noticeable performance gains (Clark et al., 2016). This inevitably made research in school- level science Question Answering (QA) hard for languages other than English due to the scarceness of resources (Clark et al., 2014; Khot et al., 2017, 2018; Bhakthavatsalam et al., 2020). ar bg de es fr hr hu it lt mk pl pt sq sr tr vi Natural Sciences Social Sciences Others Subjects Languages Geology Chemistry Physics Biology Science Politics Ethics Citizenship Geography History Philosophy Psychology Social Sociology Business & Economics Professional Forestry Landscaping Agriculture Tourism Fine Arts Informatics Islamic Studies Religion Si quitamos un electrón de un átomo de oxígeno, ¿qué obtenemos? A) Un isótopo. B) Un núcleo desnudo. C) Un ion. D) Un átomo excitado. Milyen számlák egyenlege jelenik meg a Mérlegben? A) Eredmény, forrás. B) Eszköz, forrás, eredmény. C) Eszköz, eredmény. D) Eszköz, forrás. etc. Figure 1: Properties and examples from Eχαµs. There has been a recent mini-revolution in QA, as well as in the field of Natural Language Pro- cessing (NLP) in general, due to the invention of the Transformer (Vaswani et al., 2017), and the subsequent rise of large-scale pre-trained mod- els (Peters et al., 2018; Radford et al., 2018, 2019; Devlin et al., 2019; Lan …

Content truncated to 500 words