Beyond English-Only Reading Comprehension: Experiments in Zero-shot Multilingual Transfer for Bulgarian

Proceedings of Recent Advances in Natural Language Processing, pages 447–459, Varna, Bulgaria, Sep 2–4, 2019. https://doi.org/10.26615/978-954-452-056-4_053 447 Beyond English-Only Reading Comprehension: Experiments in Zero-Shot Multilingual Transfer for Bulgarian Momchil Hardalov1 Ivan Koychev1 Preslav Nakov2 1Sofia University “St. Kliment Ohridski”, Bulgaria, 2Qatar Computing Research Institute, HBKU, Qatar, {hardalov, koychev}@fmi.uni-sofia.bg pnakov@hbku.edu.qa Abstract Recently, reading comprehension mod- els achieved near-human performance on large-scale datasets such as CoQA, MS Macro, RACE, etc.SQuAD, This is largely due to the release of pre-trained contextualized representations such as BERT and ELMo, which can be fine-tuned for the target task. Despite those ad- vances and the creation of more challeng- ing datasets, most of the work is still done for English. Here, we study the effec- tiveness of multilingual BERT fine-tuned on large-scale English datasets for reading comprehension (e.g., for RACE), and we apply it to Bulgarian multiple-choice read- ing comprehension. We propose a new dataset containing 2,221 questions from matriculation exams for twelfth grade in various subjects —history, biology, ge- ography and philosophy—, and 412 ad- ditional questions from online quizzes in history. While the quiz authors gave no relevant context, we incorporate knowl- edge from Wikipedia, retrieving docu- ments matching the combination of ques- tion + each answer option. Moreover, we experiment with different indexing and pre-training strategies. The evaluation re- sults show accuracy of 42.23%, which is well above the baseline of 24.89%. 1 Introduction The ability to answer questions is natural to hu- mans, independently of their native language, and, once learned, it can be easily transferred to another language. After understanding the question, we typically depend on our background knowledge, and on relevant information from external sources. Machines do not have the reasoning ability of hu- mans, but they are still able to learn concepts. The growing interest in teaching machines to answer questions posed in natural language has led to the introduction of various new datasets for different tasks such as reading comprehen- sion, both extractive, e.g., span-based (Nguyen et al., 2016; Trischler et al., 2017; Joshi et al., 2017; Rajpurkar et al., 2018; Reddy et al., 2019), and non-extractive, e.g., multiple-choice ques- tions (Richardson et al., 2013; Lai et al., 2017; Clark et al., 2018; Mihaylov et al., 2018; Sun et al., 2019a). Recent advances in neural network architectures, especially the raise of the Trans- former (Vaswani et al., 2017), and better contextu- alization of language models (Peters et al., 2018; Devlin et al., 2019; Radford et al., 2018; Grave et al., 2018; Howard and Ruder, 2018; Radford et al., 2019; Yang et al., 2019b; Dai et al., 2019) offered new opportunities to advance the field. Here, we investigate skill transfer from a high- resource language, i.e., English, to a low-resource one, i.e., Bulgarian, for the task of multiple-choice reading comprehension. Most previous work (Pan et al., 2018; Radford et al., 2018; Tay et al., 2018; Sun et al., 2019b) was monolingual, and a rele- vant context for each question was available a pri- ori. We take the task a step …

Content truncated to 500 words