ConditionalQA: A Complex Reading Comprehension Dataset with Conditional Answers

Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics Volume 1: Long Papers, pages 3627 - 3637 May 22-27, 2022 c⃝2022 Association for Computational Linguistics ConditionalQA: A Complex Reading Comprehension Dataset with Conditional Answers Haitian Sun School of Computer Science Carnegie Mellon University haitians@cs.cmu.edu William W. Cohen Google Research wcohen@google.com Ruslan Salakhutdinov School of Computer Science Carnegie Mellon University rsalakhu@cs.cmu.edu Abstract We describe a Question Answering (QA) dataset that contains complex questions with conditional answers, i.e. the answers are only applicable when certain conditions apply. An- swering the questions requires compositional logical reasoning across complex context. We call this dataset ConditionalQA. In addition to conditional answers, the dataset also features: (1) long context documents with information that is related in logically complex ways; (2) multi-hop questions that require compositional logical reasoning; (3) a combination of extrac- tive questions, yes/no questions, questions with multiple answers, and not-answerable ques- tions; (4) questions asked without knowing the answers. We show that ConditionalQA is chal- lenging for many of the existing QA models, especially in selecting answer conditions. We believe that this dataset will motivate further research in understanding complex documents to answer hard questions.1 1 Introduction Many reading comprehension (RC) datasets have been recently proposed (Rajpurkar et al., 2016, 2018; Kwiatkowski et al., 2019; Yang et al., 2018; Dasigi et al., 2021; Ferguson et al., 2020). In a reading comprehension task, models are provided with a document and a question and asked to find the answers. Questions in existing reading compre- hension datasets generally have a unique answer or a list of answers that are equally correct, e.g. “Who was the president of the US?” with the an- swers “George Washington”, “Thomas Jefferson”, etc. We say that these questions have deterministic answers. However, questions in the real world do not always have deterministic answers, i.e. answers to the questions are different under different con- ditions. For example, in Figure 1, the document 1https://haitian-sun.github.io/ conditionalqa/ Figure 1: An example of question and document in Condi- tionalQA dataset. The left side is a snapshot of the document discussing the eligibility of the benefit “Funeral Expense Pay- ment”. The text span “Her husband” satisfies the requirement on the “relationship with the decease” (in yellow). Text pieces in green and red are requirements that must be satisfied and thus selected as conditions for the “Yes” and “No” answers. discusses “Funeral Expense Payment” and a ques- tion asks an applicant’s eligibility. This question cannot be deterministically answered: the answer is “yes” only if “you’re arranging a funeral in the UK”, while the answer is “no”, if “... another close relative of the deceased is in work” is true. We call answers that are different under different conditions conditional answers. A conditional answer consists of an answer and a list of conditions. An answer is only true if its conditions apply. In the example above, “you are ar- ranging a funeral in the UK” is the condition for the answer “yes”. An answer can have multiple …

Content truncated to 500 words