FrugalScore: Learning Cheaper, Lighter and Faster Evaluation Metrics for Automatic Text Generation

Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics Volume 1: Long Papers, pages 1305 - 1318 May 22-27, 2022 c⃝2022 Association for Computational Linguistics FrugalScore: Learning Cheaper, Lighter and Faster Evaluation Metrics for Automatic Text Generation Moussa Kamal Eddine1, Guokan Shang2*, Antoine J.-P. Tixier1*, Michalis Vazirgiannis1,3 1École Polytechnique, 2Linagora, 3AUEB Abstract Fast and reliable evaluation metrics are key to R&D progress. While traditional natural lan- guage generation metrics are fast, they are not very reliable. Conversely, new metrics based on large pretrained language models are much more reliable, but require significant computa- tional resources. In this paper, we propose Fru- galScore, an approach to learn a fixed, low cost version of any expensive NLG metric, while retaining most of its original performance. Ex- periments with BERTScore and MoverScore on summarization and translation show that Fru- galScore is on par with the original metrics (and sometimes better), while having several orders of magnitude less parameters and running sev- eral times faster. On average over all learned metrics, tasks, and variants, FrugalScore re- tains 96.8% of the performance, runs 24 times faster, and has 35 times less parameters than the original metrics. We make our trained met- rics publicly available1 and easily accessible via Hugging Face, to benefit the entire NLP community and in particular researchers and practitioners with limited resources. 1 Introduction Automatic evaluation metrics are the only way to monitor the training of, evaluate, and compare across models in a systematic, large-scale way, and are thus a critical component of the research and development ecosystem in machine learning. To get adopted in practice, evaluation metrics need to be both reliable and affordable, i.e., fast and easy to compute. While some metrics meet these criteria, such as precision and recall in information retrieval, root mean square error in regression, etc., finding suit- able metrics is still an open problem in the field of Natural Language Generation (NLG) (Novikova et al., 2017). *Equal contribution 1https://github.com/moussaKam/FrugalScore Indeed, historical n-gram matching metrics such as ROUGE (Lin, 2004) for summarization, BLEU (Papineni et al., 2002) and METEOR (Banerjee and Lavie, 2005) for translation, while affordable, are not very reliable, as they are based on surface-form matching only, i.e., lexical similarity, and have thus no sense of semantic similarity. For instance, it makes little sense to use ROUGE for the evaluation of abstractive summarization systems (which are becoming the norm), or whenever the generated text paraphrases the original text. Following the advent of transfer learning in NLP, new NLG metrics based on large pretrained lan- guage models have recently been proposed, such as BERTScore (Zhang et al., 2019) and MoverScore (Zhao et al., 2019). By relying on contextual em- beddings, these metrics capture semantics and are therefore much more reliable. However, due to the sheer size of the underlying models, these metrics pose environmental issues (Strubell et al., 2019), take time to compute, and require access to sig- nificant computational resources, so they are not accessible by everyone in the NLP community. …

Content truncated to 500 words