FQuAD: French Question Answering Dataset

Findings of the Association for Computational Linguistics: EMNLP 2020, pages 1193–1208 November 16 - 20, 2020. c⃝2020 Association for Computational Linguistics 1193 FQuAD: French Question Answering Dataset Martin d’Hoffschmidt, Wacim Belblidia, Quentin Heinrich, Tom Brendlé Illuin Technology Paris, France {martin, wacim, quentin, tom}@illuin.tech Maxime Vidal ETH Zurich mvidal@student.ethz.ch Abstract Recent advances in the field of language modeling have improved state-of-the-art re- sults on many Natural Language Processing tasks. Among them, Reading Comprehen- sion has made significant progress over the past few years. However, most results are re- ported in English since labeled resources avail- able in other languages, such as French, re- main scarce. In the present work, we intro- duce the French Question Answering Dataset (FQuAD). FQuAD is a French Native Read- ing Comprehension dataset of questions and answers on a set of Wikipedia articles that consists of 25,000+ samples for the 1.0 ver- sion and 60,000+ samples for the 1.1 version. We train a baseline model which achieves an F1 score of 92.2 and an exact match ra- tio of 82.1 on the test set. In an effort to track the progress of French Question An- swering models we propose a leaderboard and we have made the 1.0 version of our dataset freely available at https://illuin-tech. github.io/FQuAD-explorer/. 1 Introduction Current progress in language modeling has led to increasingly successful results on various Nat- ural Language Processing (NLP) tasks. This is namely the case of the Reading Comprehension task (Richardson et al., 2013). However, Reading Comprehension datasets are costly and difficult to collect and are essentially native English datasets. Indeed, datasets such as SQuAD1.1 (Rajpurkar et al., 2016), SQuAD2.0 (Rajpurkar et al., 2018), or CoQA (Reddy et al., 2018) have fostered important and impressive progress for English Question An- swering models over the past few years. The lack of native language annotated datasets apart from English is one of the main reasons why the devel- opment of language specific Question Answering models is lagging behind and this is namely the case for French. In order to fill the gap for the French lan- guage, we introduce a French Reading Compre- hension dataset similar to SQuAD1.1. The dataset consists of French native questions and answers samples annotated by a team of university stu- dents. The dataset comes in two versions. First FQuAD1.0, containing over 25,000+ samples. Sec- ond, FQuAD1.1 containing over 60,000+ sam- ples. The 35,000+ additional samples have been annotated with more demanding guidelines to strengthen complexity of the data and model to make the task harder. More specifically, the train- ing, development, and test sets of FQuAD1.0 con- tain respectively 20,703, 3,188, and 2,189 samples. And the training, development, and test sets of FQuAD1.1 contain respectively 50,741, 5,668, and 5,594 samples. In order to evaluate the FQuAD dataset, we per- form various experiments by fine-tuning BERT based Question Answering models on both ver- sions of the FQuAD dataset. The experiments in- volve the fine-tuning of French monolingual model CamemBERT (Martin et al., 2019), and multilin- gual models mBERT …

Content truncated to 500 words