Evaluating Open-Domain Question Answering in the Era of Large Language Models

Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics Volume 1: Long Papers, pages 5591–5606 July 9-14, 2023 ©2023 Association for Computational Linguistics Evaluating Open-Domain Question Answering in the Era of Large Language Models Ehsan Kamalloo ♢ ♣ Nouha Dziri ♠ Charles L. A. Clarke ♣ Davood Rafiei ♢ ♢ University of Alberta ♣ University of Waterloo ♠ Allen Institute for Artificial Intelligence ekamalloo@uwaterloo.ca Abstract Lexical matching remains the de facto eval- uation method for open-domain question an- swering (QA). Unfortunately, lexical matching fails completely when a plausible candidate an- swer does not appear in the list of gold answers, which is increasingly the case as we shift from extractive to generative models. The recent suc- cess of large language models (LLMs) for QA aggravates lexical matching failures since can- didate answers become longer, thereby making matching with the gold answers even more chal- lenging. Without accurate evaluation, the true progress in open-domain QA remains unknown. In this paper, we conduct a thorough analysis of various open-domain QA models, including LLMs, by manually evaluating their answers on a subset of NQ-OPEN, a popular benchmark. Our assessments reveal that while the true per- formance of all models is significantly under- estimated, the performance of the InstructGPT (zero-shot) LLM increases by nearly +60%, making it on par with existing top models, and the InstructGPT (few-shot) model actually achieves a new state-of-the-art on NQ-OPEN. We also find that more than 50% of lexical matching failures are attributed to semantically equivalent answers. We further demonstrate that regex matching ranks QA models consis- tent with human judgments, although still suf- fering from unnecessary strictness. Finally, we demonstrate that automated evaluation models are a reasonable surrogate for lexical matching in some circumstances, but not for long-form answers generated by LLMs. The automated models struggle in detecting hallucinations in LLM answers and are thus unable to evaluate LLMs. At this time, there appears to be no substitute for human evaluation.1 1 Introduction Reliable benchmarks have been a bedrock to mea- suring progress in open-domain QA, the task of an- 1Code and data are released at https://github.com/ ehsk/OpenQA-eval. What was the city of Beijing previously known as? Peking Jicheng Lexical Match QA Model Who won the Oscar for best actor in 1975? QA Model Art Carney Jack Nicholson won the Oscar for Best Actor in 1975 for his performance in One Flew Over the Cuckoo's Nest. Lexical Match Figure 1: Examples of failures in open-domain QA eval- uation. Top: Jicheng is a credible answer although not present in the list of gold answers. Existing automated evaluation mechanisms fail to identify it as correct. Bot- tom: A seemingly correct but unattributable answer from InstructGPT (Ouyang et al., 2022) for which auto- matic evaluation goes astray. swering information-seeking questions over a mas- sive text corpus. In recent years, we have seen great strides in open-domain QA by novel models (Chen et al. 2017; Wang et al. 2018; Clark and Gardner 2018; Lee et al. 2019; Asai et …

Content truncated to 500 words