NLG-Metricverse: An End-to-End Library for Evaluating Natural Language Generation

Proceedings of the 29th International Conference on Computational Linguistics, pages 3465–3479 October 12–17, 2022. 3465 NLG-METRICVERSE: An End-to-End Library for Evaluating Natural Language Generation Giacomo Frisoni, Antonella Carbonaro, Gianluca Moro, Andrea Zammarchi and Marco Avagnano Department of Computer Science and Engineering (DISI) University of Bologna, Via dell’Università 50, I-47522 Cesena, Italy {giacomo.frisoni,antonella.carbonaro,gianluca.moro}@unibo.it {andrea.zammarchi3, marco.avagnano}@studio.unibo.it Abstract Driven by deep learning breakthroughs, natu- ral language generation (NLG) models have been at the center of steady progress in the last few years, with a ubiquitous task influ- ence. However, since our ability to generate human-indistinguishable artificial text lags be- hind our capacity to assess it, it is paramount to develop and apply even better automatic evalu- ation metrics. To facilitate researchers to judge the effectiveness of their models broadly, we introduce NLG-METRICVERSE—an end-to- end open-source library for NLG evaluation based on Python. Our framework provides a living collection of NLG metrics in a unified and easy-to-use environment, supplying tools to efficiently apply, analyze, compare, and visu- alize them. This includes (i) the extensive sup- port to heterogeneous automatic metrics with n-arity management, (ii) the meta-evaluation upon individual performance, metric-metric and metric-human correlations, (iii) graphical interpretations for helping humans better gain score intuitions, (iv) formal categorization and convenient documentation to accelerate met- rics understanding. NLG-METRICVERSE aims to increase the comparability and replicability of NLG research, hopefully stimulating new contributions in the area. 1 1 Introduction Natural language generation (NLG) is a sub-field of natural language processing (NLP) concerned with automatically generating human-understandable text from input data, like prompts, tables, graphs, and images. Remarkably, the ability of a machine to produce text indistinguishable from that written by humans is a pre-requisite for Artificial General Intelligence (AGI)—the holy grail of AI. Recent ad- vancements in deep learning have yielded tremen- dous improvements in the NLP sector, making 1The code is publicly available at https://github. com/disi-unibo-nlp/nlg-metricverse Metrics Scores Visualization Meta-Evaluation K Scorer metric-performance metric-metric Standard Benchmarks Human Scores metric- human Predictions References 1:1 1:N N:M Contexts Figure 1: NLG-METRICVERSE operational represen- tation. Dashed boxes denote optionality. A set of auto- matic metrics is selected to build a Scorer object, concur- rently appliable to contexts, predictions, and references with arbitrary n-arity. A Meta-Evaluation module al- lows to inspect metrics’ performance on the input data or standard benchmarks. Finally, a Visualization mod- ule can be applied to overcome opacity and understand metric-specific scoring processes. NLG the object of fast-growing interest from the re- search community, as aptly demonstrated by GPT-3 (Brown et al., 2020). Pre-trained language mod- els with transformer-based architectures (Kalyan et al., 2021) continue to push the envelope with un- precedented performance and encourage more and more applications. Indeed, today NLG includes a wide variety of tasks, such as machine translation, single/multi-document summarization, data-to-text, text-to-text, dialogue generation, free-form ques- tion answering, and image/video captioning (Gatt and Krahmer, 2018). As NLG models get better over time, accurately evaluating them is becoming an increasingly press- ing priority for tracking progress in the area and convincingly recognizing state-of-the-art systems. However, …

Content truncated to 500 words