Cross-lingual and Cross-domain Evaluation of Machine Reading Comprehension with Squad and CALOR-Quest Corpora

Proceedings of the 12th Conference on Language Resources and Evaluation (LREC 2020), pages 5491–5497 Marseille, 11–16 May 2020 c⃝ European Language Resources Association (ELRA), licensed under CC-BY-NC 5491 Cross-lingual and cross-domain evaluation of Machine Reading Comprehension with Squad and CALOR-Quest corpora Delphine Charlet1, G´eraldine Damnati1, Fr´ed´eric B´echet2, Gabriel Marzinotto1,2, Johannes Heinecke1 (1) Orange Labs, Lannion (2) Aix-Marseille Univ, Universit´e de Toulon, CNRS, LIS, Marseille, France (1) {first.last}@lis-lab.fr (2) {first.last}@orange.com Abstract Machine Reading received recently a lot of attention thanks to both the availability of very large corpora such as SQuAD or MS MARCO containing triplets (document, question, answer), and the introduction of Transformer Language Models such as BERT which obtains excellent results, even matching human performance according to the SQuAD leaderboard. One of the key features of Transformer Models is their ability to be jointly trained across multiple languages, using a shared subword vocabulary, leading to the construction of cross-lingual lexical representations. This feature has been used recently to perform zero-shot cross-lingual experiments where a multilingual BERT model fine-tuned on a machine reading comprehension task exclusively for English was directly applied to Chinese and French documents with interesting performance. In this paper we study the cross-language and cross-domain capabilities of BERT on a Machine Reading Comprehension task on two corpora: SQuAD and a new French Machine Reading dataset, called CALOR-QUEST . The semantic annotation available on CALOR-QUEST allows us to give a detailed analysis on the kind of questions that are properly handled through the cross-language process. Keywords: Machine Reading Comprehension, cross-lingual, FrameNet 1. Introduction The Machine Reading Comprehension (MRC) task re- ceived recently much attention thanks to both the availabil- ity of very large corpora such as SQuAD or MS MARCO containing triplets (document, question, answer), and the introduction of Transformer Language Models such as BERT which obtains excellent results, even matching hu- man performance according to the SQuAD leaderboard. One of the key features of the Transformer Models is their ability to be jointly trained across multiple languages, us- ing a shared subword vocabulary, leading to the construc- tion of cross-lingual lexical representations. This feature has been used recently to perform zero-shot cross-lingual experiments where a multilingual BERT model fine-tuned on a MRC task exclusively for English was directly applied to Chinese and French documents with promising perfor- mance. This study follows this path by comparing the impact of cross-language and cross-domain mismatch between train- ing and testing corpora on a MRC task with BERT. More- over, we propose to qualify each question according to a taxonomy and study which kind of questions are robust to cross-lingual transfer. We perform our study on SQuAD as well as on a new French Machine Reading dataset, called CALOR-QUEST , which contains texts on other topics than SQuAD. One of the particularities of CALOR-QUEST is the fact that it was built on a semantically annotated corpus, with a Berkeley FrameNet model. For each question, a semantic descrip- tion of the expected answer, as well as the question trigger is provided, allowing …

Content truncated to 500 words