Answer Consolidation: Formulation and Benchmarking

Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 4314 - 4325 July 10-15, 2022 ©2022 Association for Computational Linguistics Answer Consolidation: Formulation and Benchmarking Wenxuan Zhou1∗, Qiang Ning2, Heba Elfardy2, Kevin Small2, Muhao Chen1 1University of Southern California, 2Amazon {zhouwenx,muhaoche}@usc.edu {qning,helfardy,smakevin}@amazon.com Abstract Current question answering (QA) systems pri- marily consider the single-answer scenario, where each question is assumed to be paired with one correct answer. However, in many real-world QA applications, multiple answer scenarios arise where consolidating answers into a comprehensive and non-redundant set of answers is a more efficient user interface. In this paper, we formulate the problem of an- swer consolidation, where answers are parti- tioned into multiple groups, each representing different aspects of the answer set. Then, given this partitioning, a comprehensive and non- redundant set of answers can be constructed by picking one answer from each group. To initiate research on answer consolidation, we construct a dataset consisting of 4,699 ques- tions and 24,006 sentences and evaluate multi- ple models. Despite a promising performance achieved by the best-performing supervised models, we still believe this task has room for further improvements.1 1 Introduction Open-domain question answering (QA) sys- tems (Voorhees, 1999) aim to answer natural lan- guage questions using large collections of refer- ence documents, contributing to real-world appli- cations such as intelligent virtual assistants and search engines. Current QA systems (Zhu et al., 2021) usually adopt a three-stage pipeline consist- ing of: (1) a passage retriever (Yang et al., 2019; Karpukhin et al., 2020) that selects a small set of passages relevant to the question, (2) a machine reader that examines the retrieved passages and ex- tracts (Wang et al., 2017, 2019) or abstracts (Lewis et al., 2020) the candidate answers, and (3) an an- swer reranker (Wang et al., 2018; Kratzwald et al., ∗This work was conducted when the first author was doing an internship at Amazon. 1The contributed resources and implementation are avail- able at https://github.com/amazon-research/ question-answer-consolidation. 2019) that fuses features from previous stages to ei- ther select one final answer or return the top-ranked answer from the previous stage. Current QA research (Joshi et al., 2017; Kwiatkowski et al., 2019) primarily examines the case where each question is assumed to have a sin- gle correct answer. However, in practice, many questions can have multiple correct answers. For example, the question “Is coffee good for your health?” can be answered with respect to different aspects (e.g., “coffee can help you with weight loss”, “coffee can cause insomnia and restlessness”). To correctly identify different aspects of answers to the same question while mitigating aspect-level redun- dancy, it is important to consolidate the answers. Answer consolidation is particularly desirable for applications such as intelligent assistants, where re- sponses are desired to be both comprehensive and concise. Additionally, in scenarios where QA is used for knowledge extraction (Bhutani et al., 2019; Du and Cardie, 2020) or claim verification (Yin and Roth, 2018; Zhang et al., …

Content truncated to 500 words