Ditch the Gold Standard: Re-evaluating Conversational Question Answering

Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics Volume 1: Long Papers, pages 8074 - 8085 May 22-27, 2022 c⃝2022 Association for Computational Linguistics Ditch the Gold Standard: Re-evaluating Conversational Question Answering Huihan Li∗ Tianyu Gao∗ Manan Goenka Danqi Chen Department of Computer Science, Princeton University {huihanl,tianyug,mgoenka,danqic}@princeton.edu Abstract Conversational question answering aims to provide natural-language answers to users in information-seeking conversations. Existing conversational QA benchmarks compare mod- els with pre-collected human-human conver- sations, using ground-truth answers provided in conversational history. It remains unclear whether we can rely on this static evalua- tion for model development and whether cur- rent systems can well generalize to real-world human-machine conversations. In this work, we conduct the first large-scale human evalua- tion of state-of-the-art conversational QA sys- tems, where human evaluators converse with models and judge the correctness of their an- swers. We find that the distribution of human- machine conversations differs drastically from that of human-human conversations, and there is a disagreement between human and gold- history evaluation in terms of model rank- ing. We further investigate how to improve automatic evaluations, and propose a question rewriting mechanism based on predicted his- tory, which better correlates with human judg- ments. Finally, we analyze the impact of var- ious modeling strategies and discuss future di- rections towards building better conversational question answering systems.1 1 Introduction Conversational question answering aims to build machines to answer questions in conversations and has the promise to revolutionize the way humans in- teract with machines for information seeking. With recent development of large-scale datasets (Choi et al., 2018; Saeidi et al., 2018; Reddy et al., 2019; Campos et al., 2020), rapid progress has been made in better modeling of conversational QA systems. Current conversational QA datasets are collected by crowdsourcing human-human conversations, *The first two authors contributed equally. 1Our data and code are publicly available at https:// github.com/princeton-nlp/EvalConvQA. where the questioner asks questions about a specific topic, and the answerer provides answers based on an evidence passage and the conversational history. When evaluating conversational QA systems, a set of held-out conversations are used for asking mod- els questions in turn. Since the evaluation builds on pre-collected conversations, the gold history of the conversation is always provided, regardless of models’ actual predictions (Figure 1(b)). Although current systems achieve near-human F1 scores on this static evaluation, it is questionable whether this can faithfully reflect models’ true performance in real-world applications. To what extent do human- machine conversations deviate from human-human conversations? What will happen if models have no access to ground-truth answers in a conversation? To answer these questions and better understand the performance of conversational QA systems, we carry out the first large-scale human evalua- tion with four state-of-the-art models on the QuAC dataset (Choi et al., 2018) by having human eval- uators converse with the models and judge the correctness of their answers. We collected 1,446 human-machine conversations in total, with 15,059 question-answer pairs. Through careful analy- sis, we notice a significant distribution shift from …

Content truncated to 500 words