A Critical Evaluation of Evaluations for Long-form Question Answering

Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics Volume 1: Long Papers, pages 3225–3245 July 9-14, 2023 ©2023 Association for Computational Linguistics A Critical Evaluation of Evaluations for Long-form Question Answering Fangyuan Xu♢∗ Yixiao Song♡∗ Mohit Iyyer♡ Eunsol Choi♢ ♢The University of Texas at Austin, ♡University of Massachusetts Amherst {fangyuan, eunsol}@utexas.edu yixiaosong@umass.edu, miyyer@cs.umass.edu Abstract Long-form question answering (LFQA) en- ables answering a wide range of questions, but its flexibility poses enormous challenges for evaluation. We perform the first targeted study of the evaluation of long-form answers, cover- ing both human and automatic evaluation prac- tices. We hire domain experts in seven areas to provide preference judgments over pairs of answers, along with free-form justifications for their choices. We present a careful analysis of experts’ evaluation, which focuses on new aspects such as the comprehensiveness of the answer. Next, we examine automatic text gen- eration metrics, finding that no existing met- rics are predictive of human preference judg- ments. However, some metrics correlate with fine-grained aspects of answers (e.g., coher- ence). We encourage future work to move away from a single “overall score” of the answer and adopt a multi-faceted evaluation, targeting as- pects such as factuality and completeness. We publicly release all of our annotations and code to spur future work into LFQA evaluation.1 1 Introduction Long-form question answering (Fan et al., 2019; Krishna et al., 2021; Nakano et al., 2021; Su et al., 2022, henceforth LFQA), an emerging research area within QA, requires systems to generate long and complex answers to questions by leveraging large language models and evidence document re- trievers. While remarkable strides have been made in LFQA model development, the current state of LFQA evaluation is dire: most prior papers use a combination of crowdsourced human annotations and simple string-matching metrics (e.g., ROUGE). We present the first study of the evaluation of long- form answers, exploring both human and automatic evaluation protocols to better understand how we should evaluate LFQA moving forward. ∗Equal contribution. 1https://github.com/carriex/lfqa_eval Human evaluation: In most prior human LFQA evaluations (Krishna et al., 2021; Nakano et al., 2021), crowd annotators are given a question, two candidate answers, and (optionally) evidence doc- uments, and they are asked to identify the better answer. However, crowdworkers do not necessarily have the expertise or background knowledge to reli- ably judge properties such as factuality (Gillick and Liu, 2010; Iskender et al., 2020). Thus, we hire do- main experts in seven different fields (e.g., biology, economics) to perform the same answer preference task and additionally provide detailed justifications as to why they chose a particular answer. Analyz- ing their justifications reveals that experts consider properties such as completeness and factuality to be more decisive than surface-level aspects (e.g., conciseness and level of detail) on which crowd- workers tend to fixate. Additionally, even experts often disagree with each other about which answer is better; this disagreement stems from valuing fine- grained answer properties differently. Automatic evaluation: As human evaluation is slow and expensive, developing a …

Content truncated to 500 words