The statistical advantage of automatic NLG metrics at the system level

Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing, pages 6840–6854 August 1–6, 2021. ©2021 Association for Computational Linguistics 6840 The statistical advantage of automatic NLG metrics at the system level Johnny Tian-Zheng Wei Robin Jia Department of Computer Science, University of Southern California {jtwei, robinjia}@usc.edu Abstract Estimating the expected output quality of gen- eration systems is central to NLG. This paper qualifies the notion that automatic metrics are not as good as humans in estimating system- level quality. Statistically, humans are unbi- ased, high variance estimators, while metrics are biased, low variance estimators. We com- pare these estimators by their error in pairwise prediction (which generation system is better?) using the bootstrap. Measuring this error is complicated: predictions are evaluated against noisy, human predicted labels instead of the ground truth, and metric predictions fluctuate based on the test sets they were calculated on. By applying a bias-variance-noise decomposi- tion, we adjust this error to a noise-free, in- finite test set setting. Our analysis compares the adjusted error of metrics to humans and a derived, perfect segment-level annotator, both of which are unbiased estimators dependent on the number of judgments collected. In MT, we identify two settings where metrics outperform humans due to a statistical advantage in vari- ance: when the number of human judgments used is small, and when the quality difference between compared systems is small.1 1 Introduction Automatic metrics are involved in many develop- mental settings for natural language generation (NLG) systems. In machine translation (MT), met- rics like BLEU (Papineni et al., 2002) enable set- tings where the amount of human effort required would be infeasible, such as architecture or hyper- parameter search (Britz et al., 2017). As objec- tive, reproducible quantities, BLEU scores facil- itate cross-paper comparisons (Post, 2018). His- torically, progress in MT has been attributed to its 1The data and code to reproduce our analyses are available at https://github.com/johntzwei/ metric-statistical-advantage. Bias δH S,S′ (True difference) 0 Dist. of estimator Metric (� δM S,S′) Human (� δH S,S′) Figure 1: Distribution of estimators for the true differ- ence in system quality δH S,S′ between two generation systems (for illustrative purposes). Notation is defined in §2.3. An estimate incurs prediction error if its sign is opposite to the true difference. While humans pro- vide an unbiased estimator of the difference, a biased estimator derived from a metric can have a smaller er- ror probability (shaded areas) due to its lower variance. Evidence supporting the illustration can be found in §5. use (Callison-Burch et al., 2006). Metrics are an active research area in many NLG subfields, in- cluding summarization (Lin, 2004), dialogue (Tao et al., 2018), and image captioning (Anderson et al., 2016), which seek to realize the goal of quick and reliable automatic evaluation. In all these subfields, the primary goal when con- ducting evaluation is typically to compare NLG systems. Both human annotators and automatic metrics produce segment-level scores, i.e., scores …

Content truncated to 500 words