CoQA: A Conversational Question Answering Challenge

CoQA: A Conversational Question Answering Challenge Siva Reddy∗ Danqi Chen∗ Christopher D. Manning Computer Science Department Stanford University {sivar,danqi,manning}@cs.stanford.edu Abstract Humans gather information through conver- sations involving a series of interconnected questions and answers. For machines to assist in information gathering, it is therefore essen- tial to enable them to answer conversational questions. We introduce CoQA, a novel data- set for building Conversational Question An- swering systems. Our dataset contains 127k questions with answers, obtained from 8k conversations about text passages from seven diverse domains. The questions are conver- sational, and the answers are free-form text with their corresponding evidence highlighted in the passage. We analyze CoQA in depth and show that conversational questions have challenging phenomena not present in existing reading comprehension datasets (e.g., coref- erence and pragmatic reasoning). We evaluate strong dialogue and reading comprehension models on CoQA. The best system obtains an F1 score of 65.4%, which is 23.4 points behind human performance (88.8%), indicating that there is ample room for improvement. We present CoQA as a challenge to the commu- nity at https://stanfordnlp.github. io/coqa. 1 Introduction We ask other people a question to either seek or test their knowledge about a subject. Depending on their answer, we follow up with another question and their second answer builds on what has already been discussed. This incremental aspect makes human conversations succinct. An inability to build and maintain common ground in this way is part of why virtual assistants usually don’t seem like competent conversational partners. In this ∗The first two authors contributed equally. paper, we introduce CoQA,1 a Conversational Question Answering dataset for measuring the ability of machines to participate in a question- answering style conversation. In CoQA, a machine has to understand a text passage and answer a series of questions that appear in a conversation. We develop CoQA with three main goals in mind. The first concerns the nature of questions in a human conversation. Figure 1 shows a conver- sation between two humans who are reading a passage, one acting as a questioner and the other as an answerer. In this conversation, every question after the first is dependent on the conversation history. For instance, Q5 (Who?) is only a single word and is impossible to answer without knowing what has already been said. Posing short questions is an effective human conversation strategy, but such questions are difficult for machines to parse. As is well known, state-of-the-art models rely heavily on lexical similarity between a question and a passage (Chen et al., 2016; Weissenborn et al., 2017). At present, there are no large- scale reading comprehension datasets that contain questions that depend on a conversation history (see Table 1) and this is what CoQA is mainly developed for.2 The second goal of CoQA is to ensure the naturalness of answers in a conversation. Many existingQAdatasetsrestrictanswersto contiguous text spans in a given passage (Table 1). Such answers are not always natural—for example, there is no span-based answer to Q4 (How many?) in Figure 1. In …

Content truncated to 500 words