Read and Reason with MuSeRC and RuCoS: Datasets for Machine Reading Comprehension for Russian

Proceedings of the 28th International Conference on Computational Linguistics, pages 6481–6497 Barcelona, Spain (Online), December 8-13, 2020 6481 Read and Reason with MuSeRC and RuCoS: Datasets for Machine Reading Comprehension for Russian Alena Fenogenova1, Vladislav Mikhailov1,2, Denis Shevelev1 1 Sberbank / Moscow, Russia 2 National Research University Higher School of Economics / Moscow, Russia {Fenogenova.A.S, Mikhaylov.V.Nikola, Shevelev.D.M}@sberbank.ru Abstract The paper introduces two Russian machine reading comprehension (MRC) datasets, called MuSeRC and RuCoS, which require reasoning over multiple sentences and commonsense knowl- edge to infer the answer. The former follows the design of MultiRC, while the latter is a coun- terpart of the ReCoRD dataset. The datasets are included in RussianSuperGLUE, the Russian general language understanding benchmark. We provide a comparative analysis and demonstrate that the proposed tasks are relatively more complex as compared to the original ones for English. Besides, performance results of human solvers and BERT-based models show that MuSeRC and RuCoS represent a challenge for recent advanced neural models. We thus hope to facili- tate research in the field of MRC for Russian and prompt the study of multi-hop reasoning in a cross-lingual scenario. 1 Introduction Machine reading comprehension (MRC) is a central task in natural language processing that simulates a human ability to read a text and provide the correct answer for a given question. Therefore, it requires general language understanding, knowledge about the world, and interpretive reasoning. The task has been widely explored for the English language targeting different aspects of reading comprehension (Hermann et al., 2015; Hill et al., 2015; Rajpurkar et al., 2016; Trischler et al., 2016; Joshi et al., 2017; Rajpurkar et al., 2018). Recently, the paradigm has shifted towards a more complex setting, where the model is tested to infer the answer based on interpretive analysis of text (Lai et al., 2017), reasoning over multiple documents (Yang et al., 2018) or joint natural language inference and commonsense reasoning (Zellers et al., 2018). However, MRC in languages other than English, specifically Russian, has not been well-addressed primarily due to the lack of high-quality and large-scale datasets. Recognizing this need, several cross-lingual machine reading datasets have been constructed (Asai et al., 2018; Liu et al., 2019; Lewis et al., 2019b; Artetxe et al., 2019; Clark et al., 2020) with a few of them being a part of cross-lingual benchmarks such as XGLUE (Liang et al., 2020) and XTREME (Hu et al., 2020). Though allowing to evaluate the current state of language transferring methods, these cover only a small number of languages, use a back-translation technique for the assembly or combine data from different annotation schemes. Surprisingly, little prior research has been devoted to the task of MRC for Russian, and no at- tempts have been made to explore the subject in multi-hop and commonsense reasoning scenarios. SberQuAD (Efimov et al., 2019) is the only one Russian MRC dataset designed as a competition chal- lenge analogous to SQuAD (Rajpurkar et al., 2016). The task is formalized as an extractive reading comprehension, where the answer to …

Content truncated to 500 words