Synthetic Data Generation for Multilingual Domain-Adaptable Question Answering Systems

Synthetic Data Generation for Multilingual Domain-Adaptable Question Answering Systems Alina Kramchaninova, Arne Defauw CrossLang Kerkstraat 106, 9050 Gentbrugge, Belgium {firstname.lastname}@crosslang.com Abstract Deep learning models have significantly advanced the state of the art of question answering systems. However, the majority of datasets available for training such mod- els have been annotated by humans, are open-domain, and are composed primarily in English. To deal with these limitations, we introduce a pipeline that creates syn- thetic data from natural text. To illustrate the domain-adaptability of our approach, as well as its multilingual potential, we use our pipeline to obtain synthetic data in En- glish and Dutch. We combine the synthetic data with non-synthetic data (SQuAD 2.0) and fine-tune multilingual BERT models on the question answering task. Models trained with synthetically augmented data demonstrate a clear improvement in per- formance when evaluated on the domain- specific test set, compared to the models trained exclusively on SQuAD 2.0. We ex- pect our work to be beneficial for training domain-specific question-answering sys- tems when the amount of available data is limited. 1 Introduction Recent advances in tackling the problem of ques- tion answering (QA) rely on large-scale, open- domain datasets (Bartolo et al., 2021), annotated by humans and composed primarily in English (e.g. SQuAD 1.0 (Rajpurkar et al., 2016), and SQuAD 2.0 (Rajpurkar et al., 2018)). Despite © 2022 The authors. This article is licensed under a Creative Commons 3.0 licence, no derivative works, attribution, CC- BY-ND. some indications of poor robustness and gener- alisation (Bartolo et al., 2021), models trained on such datasets are capable of providing topic- agnostic, general-purpose assistance to their users (Ruder and Sil, 2021). Nevertheless, most industrial applications of QA systems are domain-specific, and often need to be able to operate in multilingual environ- ments. Data collection and manual composition of datasets for each domain and language is most def- initely a laborious task, not to mention that certain domains are of little academic or commercial inter- est and are only of use for some low-resource com- munities (Rogers et al., 2021). Moreover, while the current synthetic data generation systems focus on augmenting QA data in the SQuAD format,1 lit- tle research has been done on either the generation of synthetic data from natural plain text, or in mul- tiple languages. Furthermore, most machine reading comprehen- sion (MRC) benchmarks focus primarily on the creation of questions with multi-word factoid an- swers (e.g. SQuAD 2.0 pairs each factoid question with a Wikipedia paragraph), as well as unanswer- able questions (Liu et al., 2020). However, in a real-world scenario, a QA system should ideally be able to provide a response on semantically com- plex questions such as “I am an EU citizen living in the UK. What changes for me after Brexit?”, and questions containing grammar and spelling errors (e.g. questions asked by a non-native speaker, or containing mistakes caused by dyslexia). In this work, we introduce a domain-adaptable end-to-end pipeline for generic synthetic data generation that requires no manual textual pre- processing, …

Content truncated to 500 words