FQuAD2.0: French Question Answering and Learning When You Don’t Know

Proceedings of the 13th Conference on Language Resources and Evaluation (LREC 2022), pages 2205–2214 Marseille, 20-25 June 2022 © European Language Resources Association (ELRA), licensed under CC-BY-NC-4.0 2205 FQuAD2.0: French Question Answering and Learning When You Don’t Know Quentin Heinrich, Gautier Viaud, Wacim Belblidia Illuin Technology Paris, France {quentin.heinrich, gautier.viaud, wacim.belblidia}@illuin.tech Abstract Question Answering, including Reading Comprehension, is one of the NLP research areas that has seen significant sci- entific breakthroughs over the past few years, thanks to the concomitant advances in Language Modeling. Most of these breakthroughs, however, are centered on the English language. As a first strong initiative to bridge the gap with the French language, FQuAD1.1 was introduced in 2020: it is a French Native Reading Comprehension dataset composed of 60,000+ questions and answers samples extracted from Wikipedia articles. Nonetheless, Question Answering models trained on this dataset have a major drawback: they are not able to predict when a given question has no answer in the paragraph of interest, therefore making unreliable predictions in various industrial use-cases. We introduce FQuAD2.0, which extends FQuAD with 17,000+ unanswerable questions, annotated adversarially, in order to be similar to answerable ones. This new dataset, comprising a total of almost 80,000 questions, makes it possible to train French Question Answering models with the ability of distinguishing unanswerable questions from answerable ones. We benchmark several models with this dataset: our best model, a fine-tuned CamemBERTLARGE, achieves a F1 score of 82.3% on this classification task, and a F1 score of 83% on the Reading Comprehension task. Keywords: question answering, adversarial, french, multilingual 1. Introduction Question Answering (QA) is a central task in Natu- ral Language Understanding (NLU), with numerous industrial applications such as searching information in large corpus, extracting information from conver- sations or form filling. Amongst this domain, Read- ing Comprehension has gained a lot of traction in the past years thanks to two main factors. First, the release of numerous datasets such as SQuAD1.1 (Rajpurkar et al., 2016), SQuAD2.0 (Rajpurkar et al., 2018), BoolQ (Clark et al., 2019), CoQA (Reddy et al., 2018), Natu- ral Questions (Kwiatkowski et al., 2019), only to cite a few. Second, the progress in Language Modeling with the introduction of transformers model (Vaswani et al., 2017), leveraging self-supervised training on very large text corpus, followed by fine-tuning on a downstream task (Devlin et al., 2018). This process has now be- come a de-facto standard for most of Natural Lan- guage Processing (NLP) tasks and also contributed to the progress of state-of-the-art for most of these tasks, including Question Answering. Whilst most of these recent transformers models are English models, French language models have also been released, in particular CamemBERT (Martin et al., 2019) and FlauBERT (Le et al., 2019), as well as multilingual models such as mBERT (Pires et al., 2019) or XLM-RoBERTa (Conneau et al., 2019). These mod- els fostered the state-of-the-art in NLP for French lan- guage, allowing to benefit from this large-scale trans- fer learning mechanism. However, native French re- sources for Question …

Content truncated to 500 words