CMQA: A Dataset of Conditional Question Answering with Multiple-Span Answers

Proceedings of the 29th International Conference on Computational Linguistics, pages 1697–1707 October 12–17, 2022. 1697 CMQA: A Dataset of Conditional Question Answering with Multiple-Span Answers Yiming Ju*,1,2, Weikang Wang*,3, Yuanzhe Zhang1,2, Suncong Zheng3, Kang Liu1,2, Jun Zhao1,2, 1 National Laboratory of Pattern Recognition, Institute of Automation, CAS, Beijing, China 2 School of Artificial Intelligence, University of Chinese Academy of Sciences, Beijing, China 3 Machine Learning Platform Department, Tencent {yiming.ju, yzzhang, kliu, jzhao}@nlpr.ia.ac.cn {daxianwang, congzheng}@tencent.com Abstract Forcing the answer of the Question Answer- ing (QA) task to be a single text span might be restrictive since the answer can be multiple spans in the context. Moreover, we found that multi-span answers often appear with two char- acteristics when building the QA system for a real-world application. First, multi-span an- swers might be caused by users lacking domain knowledge and asking ambiguous questions, which makes the question need to be answered with conditions. Second, there might be hierar- chical relations among multiple answer spans. Some recent span-extraction QA datasets in- clude multi-span samples, but they only con- tain unconditional and parallel answers, which cannot be used to tackle this problem. To bridge the gap, we propose a new task: con- ditional question answering with hierarchical multi-span answers, where both the hierarchi- cal relations and the conditions need to be ex- tracted. Correspondingly, we introduce CMQA, a Conditional Multiple-span Chinese Question Answering dataset to study the new proposed task. The final release of CMQA consists of 7,861 QA pairs and 113,089 labels, where all samples contain multi-span answers, 50.4% of samples are conditional, and 56.6% of samples are hierarchical. CMQA can serve as a bench- mark to study the new proposed task and help study building QA systems for real-world appli- cations. The low performance of models drawn from related literature shows that the new pro- posed task is challenging for the community to solve. CMQA can be accessed at https: //github.com/juyiming/CMQA. 1 Introduction Question answering (QA) is a challenging bench- mark task, which can drive the development of nat- ural language understanding (NLU) methods and has significant utility to users (Kwiatkowski et al., * means both authors contributed equally to this research. 2019). This research area has made significant progress with many sizable datasets and standard benchmarks. Notably, the span-extraction task is one of the most studied subtasks because of its wide applicability and easy evaluation characteristics. Most existing span-extraction QA datasets (Ra- jpurkar et al., 2016; Yang et al., 2018; Choi et al., 2018; Kwiatkowski et al., 2019; Chen et al., 2020) only contain single-span samples, where the an- swer is a single text span in the context. However, limiting the answer to be a single span in the con- text might be restrictive since the answer of some questions can be multiple text spans in real-world QA scenarios. Take the case in Figure 1 for ex- ample, given the question asking about ‘drugs for face acne redness’, the corresponding drugs (green and blue highlighted text spans) appear in differ- ent …

Content truncated to 500 words