Project PIAF: Building a Native French Question-Answering Dataset

Proceedings of the 12th Conference on Language Resources and Evaluation (LREC 2020), pages 5481–5490 Marseille, 11–16 May 2020 c⃝ European Language Resources Association (ELRA), licensed under CC-BY-NC 5481 Project PIAF: Building a Native French Question-Answering Dataset Rachel Keraron≡,⋆, Guillaume Lancrenon≡,‡, Mathilde Bras≡,‡, Fr´ed´eric Allary⋆, Gilles Moyse⋆, Thomas Scialom⋆⋄ Edmundo-Pavel Soriano-Morales‡, Jacopo Staiano⋆ ≡ equal contribution ⋆ reciTAL, Paris (France) ‡Etalab, DINUM, Prime Minister’s Office, Paris (France) ⋄ Sorbonne Universit´e, CNRS, LIP6, F-75005 Paris, France {rachel, frederic, gilles, thomas, jacopo}@recital.ai {guillaume.lancrenon, mathilde.bras, pavel.soriano}@data.gouv.fr Abstract Motivated by the lack of data for non-English languages, in particular for the evaluation of downstream tasks such as Question Answering, we present a participatory effort to collect a native French Question Answering Dataset. Furthermore, we describe and publicly release the annotation tool developed for our collection effort, along with the data obtained and preliminary baselines. Keywords: Question Answering, Annotation, Crowdsourcing 1. Introduction Along with the availability of massive amounts of data, the increase in computational power has in recent years al- lowed the development of Deep Learning techniques, lead- ing to significant advancements in the fields of Computer Vision (CV), and Natural Language Processing (NLP), among others. Visual information can, to some extent, be considered to generalize across cultures in many real world applications; in contrast, having to deal with languages, NLP applications are naturally bound to language specifici- ties. Over the years, the NLP community has produced sev- eral resources to tackle tasks we call, for simplicity, up- stream (such as Part-Of-Speech tagging, Dependency Pars- ing, etc.), targeting multiple languages and enabling the construction of effective automated systems. Still, for tasks we refer to as downstream, i.e. those which enable the development of value-added end products such as Ques- tion Answering (QA) or Conversational Agents, the current state-of-the-art approaches require massive amounts of an- notated data which are almost exclusively available in En- glish. Notable exceptions include Machine Translation, for which abundant parallel corpora have been built from re- sources such as the European parliamentary proceedings, or Language Modeling – which can be tackled in a self- supervised manner, hence only requiring massive amounts of text in the target language(s). Such asymmetry has re- cently been acknowledged by a resolution of the EU Parlia- ment.1 In this work, we focus on collecting Question Answering data in French in a participatory setup: we describe the data collection protocol adopted, report descriptive statistics and 1European Parliament resolution of 11 September 2018 on lan- guage equality in the digital age. http://www.europarl. europa.eu/sides/getDoc.do?pubRef=-//EP/ /TEXT+TA+P8-TA-2018-0332+0+DOC+XML+V0//EN baselines, and provide details on the implementation of the open source annotation tool we developed. Such tool al- lows volunteers to participate in crowdsourced QA dataset collection campaigns. In summary, we make the following contributions: 1. we develop and release a novel annotation tool to col- lect large-scale QA data in a participatory scenario; 2. we release a native French QA dataset; 3. we provide baselines using state-of-the-art methodolo- gies. 2. Related Work Several datasets for QA have been recently produced. The Stanford Question Answering …

Content truncated to 500 words