Can Large Language Models Be an Alternative to Human Evaluations?

Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics Volume 1: Long Papers, pages 15607–15631 July 9-14, 2023 ©2023 Association for Computational Linguistics Can Large Language Models Be an Alternative to Human Evaluation? Cheng-Han Chiang National Taiwan University, Taiwan dcml0714@gmail.com Hung-yi Lee National Taiwan University, Taiwan hungyilee@ntu.edu.tw Abstract Human evaluation is indispensable and in- evitable for assessing the quality of texts gen- erated by machine learning models or written by humans. However, human evaluation is very difficult to reproduce and its quality is noto- riously unstable, hindering fair comparisons among different natural language processing (NLP) models and algorithms. Recently, large language models (LLMs) have demonstrated exceptional performance on unseen tasks when only the task instructions are provided. In this paper, we explore if such an ability of the LLMs can be used as an alternative to human evalu- ation. We present the LLMs with the exact same instructions, samples to be evaluated, and questions used to conduct human evaluation, and then ask the LLMs to generate responses to those questions; we dub this LLM evaluation. We use human evaluation and LLM evaluation to evaluate the texts in two NLP tasks: open- ended story generation and adversarial attacks. We show that the result of LLM evaluation is consistent with the results obtained by expert human evaluation: the texts rated higher by hu- man experts are also rated higher by the LLMs. We also find that the results of LLM evalua- tion are stable over different formatting of the task instructions and the sampling algorithm used to generate the answer. We are the first to show the potential of using LLMs to assess the quality of texts and discuss the limitations and ethical considerations of LLM evaluation. 1 Introduction Human evaluation is an important method to under- stand the performance of an NLP model or algo- rithm (Guzmán et al., 2015; Gillick and Liu, 2010). We rely on human evaluation because there are cer- tain aspects of texts that are hard to evaluate using automatic evaluation metrics; thus, researchers re- sort to humans to rate the quality of the output of NLP models. While human evaluation is preva- lent and indispensable in NLP, it is notoriously unstable (Gillick and Liu, 2010; Clark et al., 2021). Karpinska et al. (2021) has shown that the quality of workforces in human evaluation can have a detri- mental effect on the evaluation result, making it impossible to compare the performance among dif- ferent systems. Reproducibility is another issue in human evaluation since it is hard to recruit the same human evaluators and rerun the same evaluation. Even if the same workers are recruited, the workers that have seen the task before are likely to produce a different evaluation result the next time because they have already done the task. While human eval- uation is used to better assess NLP systems and has some advantages over automatic evaluation met- rics, the drawbacks of human evaluation somewhat make it difficult to reliably evaluate NLP systems. …

Content truncated to 500 words