A Qualitative Comparison of CoQA, SQuAD 2.0 and QuAC

Proceedings of NAACL-HLT 2019, pages 2318–2323 Minneapolis, Minnesota, June 2 - June 7, 2019. c⃝2019 Association for Computational Linguistics 2318 A Qualitative Comparison of CoQA, SQuAD 2.0 and QuAC Mark Yatskar Allen Institute for Artificial Intelligence marky@allenai.org Abstract We compare three new datasets for question answering: SQuAD 2.0, QuAC, and CoQA, along several of their new features: (1) unan- swerable questions, (2) multi-turn interactions, and (3) abstractive answers. We show that the datasets provide complementary coverage of the first two aspects, but weak coverage of the third. Because of the datasets’ struc- tural similarity, a single extractive model can be easily adapted to any of the datasets and we show improved baseline results on both SQuAD 2.0 and CoQA. Despite the similar- ity, models trained on one dataset are inef- fective on another dataset, but we find mod- erate performance improvement through pre- training. To encourage cross-evaluation, we release code for conversion between datasets at https://github.com/my89/co-squac. 1 Introduction Question answering on textual data has served as a challenge problem for the NLP community (Voorhees, 2001; Richardson et al., 2013). With the development of large scale benchmarks and sufficiently simple evaluations (Trischler et al., 2016; Nguyen et al., 2016; Hermann et al., 2015) progress has been rapid. In recent evaluation on SQuAD (Rajpurkar et al., 2016), performance ex- ceeded that of annotators (Wang et al., 2018; Hu et al., 2017; Wang et al., 2017). In response to this development, there have been a flurry of new datasets. In this work, we analyze three such new proposed datasets, SQuAD 2.0 (Rajpurkar et al., 2018), QuAC (Choi et al., 2018), and CoQA (Reddy et al., 2018).1 In each of these datasets, crowd workers are asked to (1) produce questions about a paragraph of text (context) and (2) produce a reply 1A review of other new datasets is in the related work. by either indicating there is no answer, or provid- ing an extractive answer from the context by high- lighting one contiguous span. QuAC and CoQA contain two other features: questions are asked in the form of a dialog, where co-reference to previ- ous interactions is possible and directly answering yes/no is possible. CoQA also allows workers to edit the spans to provide abstractive answers.2 We compare these three datasets along several of their new features: (1) unanswerable questions, (2) multi-turn interactions, and (3) abstractive an- swers. Unanswerable question coverage is com- plementary among datasets; SQuAD 2.0 focuses more on questions of extreme confusion, such as false premise questions, while QuAC primar- ily focuses on missing information. QuAC and CoQA dialogs simulate different types of user be- havior: QuAC dialogs often switch topics while CoQA dialogs include more queries for details. Unfortunately, no dataset provides significant cov- erage of abstractive answers beyond yes/no an- swers, and we show that a method can achieve an extractive answer upper bound of 100 and 97.8 F1 on QuAC and CoQA , respectively. Motivated by the above analysis, we apply the baseline presented in QuAC (Choi et …

Content truncated to 500 words