WikiHowQA: A Comprehensive Benchmark for Multi-Document Non-Factoid Question Answering

Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics Volume 1: Long Papers, pages 5291–5314 July 9-14, 2023 ©2023 Association for Computational Linguistics WikiHowQA: A Comprehensive Benchmark for Multi-Document Non-Factoid Question Answering Valeriia Bolotova1 lurunchik@gmail.com Vladislav Blinov2 vladislav.a.blinov@gmail.com Sofya Filippova1 s3848979@student.rmit.edu.au Falk Scholer1 falk.scholer@rmit.edu.au Mark Sanderson1 mark.sanderson@rmit.edu.au 1RMIT University 2Ural Federal University Abstract Answering non-factoid questions (NFQA) is a challenging task, requiring passage-level an- swers that are difficult to construct and evaluate. Search engines may provide a summary of a single web page, but many questions require reasoning across multiple documents. Mean- while, modern models can generate highly co- herent and fluent, but often factually incor- rect answers that can deceive even non-expert humans. There is a critical need for high- quality resources for multi-document NFQA (MD-NFQA) to train new models and evaluate answers’ grounding and factual consistency in relation to supporting documents. To bridge this gap, we present WIKIHOWQA,1 a new multi-document NFQA benchmark built on WikiHow, a website dedicated to answer- ing “how-to” questions. The benchmark in- cludes 11,746 human-written answers along with 74,527 supporting documents. We de- scribe the unique challenges of the resource, provide strong baselines, and propose a novel human evaluation framework that utilizes high- lighted relevant supporting passages to mitigate issues such as assessor unfamiliarity with the question topic. All code and data, including the automatic code for preparing the human evaluation, are publicly available. 1 Introduction Non-factoid questions (NFQs) requiring long, passage-level answers, such as explanations or opinions, pose challenges for current question- answering systems. While a few datasets exist for NFQA (Cohen and Croft, 2016; Hashemi et al., 2019; Soleimani et al., 2021), users are currently limited to seeing a summary of the most relevant document in a snippet on a search result page (SERP), which can be insufficient for complex questions that have scattered answers across multi- ple documents or require sophisticated reasoning 1https://lurunchik.github.io/WikiHowQA/ Figure 1: An instance of proposed WIKIHOWQA to generate an answer. However, even relatively straightforward questions like “how to fix my com- puter issue” can have multiple solutions, requiring the user to manually search through multiple re- trieved documents to find the one that applies to their situation. Complex questions such as “how to feel calm and relaxed” may require an aggregated summary of the most popular methods from multi- ple relevant sources. There are also questions for which answers have yet to be written, requiring a QA system to treat relevant documents as initial sources of information and then to reason out an answer based on them. For example, the question “how to prepare to buy a house in [neighbourhood], [city]” would require the system to retrieve relevant documents about buying houses in general and spe- cific articles about the local house market and the neighbourhood, and then construct a more sophis- ticated answer through reasoning rather than just summarizing multiple documents. To address these challenges and move towards more advanced QA systems that can provide in- depth and comprehensive answers to a …

Content truncated to 500 words