NLQuAD: A Non-Factoid Long Question Answering Data Set

Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics, pages 1245–1255 April 19 - 23, 2021. ©2021 Association for Computational Linguistics 1245 NLQuAD: A Non-Factoid Long Question Answering Data Set Amir Soleimani Informatics Institute University of Amsterdam Amsterdam, The Netherlands a.soleimani@uva.nl Christof Monz Informatics Institute University of Amsterdam Amsterdam, The Netherlands c.monz@uva.nl Marcel Worring Informatics Institute University of Amsterdam Amsterdam, The Netherlands m.worring@uva.nl Abstract We introduce NLQuAD, the first data set with baseline methods for non-factoid long ques- tion answering, a task requiring document- level language understanding. In contrast to existing span detection question answering data sets, NLQuAD has non-factoid questions that are not answerable by a short span of text and demanding multiple-sentence descriptive answers and opinions. We show the limitation of the F1 score for evaluation of long answers and introduce Intersection over Union (IoU), which measures position-sensitive overlap be- tween the predicted and the target answer spans. To establish baseline performances, we compare BERT, RoBERTa, and Longformer models. Experimental results and human eval- uations show that Longformer outperforms the other architectures, but results are still far be- hind a human upper bound, leaving substan- tial room for improvements. NLQuAD’s sam- ples exceed the input limitation of most pre- trained Transformer-based models, encourag- ing future research on long sequence language models.1 1 Introduction Over the last few years, there have been remark- able improvements in the area of Machine Reading Comprehension (MRC) and open-domain Question Answering (QA) due to the availability of large scale data sets such as SQuAD (Rajpurkar et al., 2016) and pre-trained language models such as BERT (Devlin et al., 2018). Although non-factoid questions represent a large number of real-life ques- tions, current QA data sets barely cover this area. The reason is that context passages in existing QA data sets are mostly very short and questions mostly factoid, i.e., can be answered by simple facts or entities such as a person name and location (Juraf- sky and Martin, 2019). Little attention has been 1Dataset and Models: github.com/asoleimanib/NLQuAD Question: How are people coping in the lockdown? Headline: China coronavirus: Death toll rises as more cities restrict travel Document: China has widened its travel restrictions in Hubei province - the centre of the coronavirus outbreak - as the death toll climbed to 26. The restrictions will affect at least 20 million people across 10 cities, includ- ing the capital, Wuhan, where the virus emerged. On Thursday, a coronavirus patient died in northern Hebei province - making it the first death outside Hubei. [...] We now know this is not a virus that will burn out on its own and disappear. [...] And we still don’t know when people are contagious. Is it before symptoms appear, or only after severe symptoms emerge? One is signifi- cantly harder to stop spreading than the other. [...] One doctor, who requested anonymity, describes the con- ditions at a hospital in Wuhan. [...] “I was planning to stay in my apartment because I’m scared to …

Content truncated to 500 words