Reproducibility Issues for BERT-based Evaluation Metrics

Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 2965 - 2989 December 7-11, 2022 ©2022 Association for Computational Linguistics Reproducibility Issues for BERT-based Evaluation Metrics Yanran Chen1, Jonas Belouadi2, Steffen Eger2 1Computer Science Department, Technical University of Darmstadt, Germany 2NLLG, Faculty of Technology, Bielefeld University, Germany 1yanran.chen@stud.tu-darmstadt.de 2{jonas.belouadi,steffen.eger}@uni-bielefeld.de Abstract Reproducibility is of utmost concern in ma- chine learning and natural language processing (NLP). In the field of natural language genera- tion (especially machine translation), the semi- nal paper of Post (2018) has pointed out prob- lems of reproducibility of the dominant met- ric, BLEU, at the time of publication. Nowa- days, BERT-based evaluation metrics consider- ably outperform BLEU. In this paper, we ask whether results and claims from four recent BERT-based metrics can be reproduced. We find that reproduction of claims and results of- ten fails because of (i) heavy undocumented preprocessing involved in the metrics, (ii) miss- ing code and (iii) reporting weaker results for the baseline metrics. (iv) In one case, the prob- lem stems from correlating not to human scores but to a wrong column in the csv file, inflating scores by 5 points. Motivated by the impact of preprocessing, we then conduct a second study where we examine its effects more closely (for one of the metrics). We find that preprocessing can have large effects, especially for highly in- flectional languages. In this case, the effect of preprocessing may be larger than the effect of the aggregation mechanism (e.g., greedy align- ment vs. Word Mover Distance). 1 Introduction Reproducibility is a core aspect in machine learn- ing (ML) and natural language processing (NLP). It requires that claims and results of previous work can independently be reproduced and is a prereq- uisite to trustworthiness. The last few years have seen vivid interest in the topic and many issues of non-reproducibility have been pointed out, leading to claims of a “reproducibility crisis” in science (Baker, 2016). In the field of evaluation metrics for natural language generation (NLG), the sem- inal work of Post (2018) has demonstrated how different preprocessing schemes can lead to sub- stantially different results when using the dominant metric at the time, BLEU (Papineni et al., 2002). Thus, when researchers employ such different pre- processing steps (a seemingly innocuous decision), this can directly lead to reproducibility failures of (conclusions regarding) metric performances. Even though BLEU and similar lexical-overlap metrics still appear to dominate the landscape of NLG (particular MT) evaluation (Marie et al., 2021), it is obvious that metrics which measure surface level overlap are unsuitable for evalua- tion, especially for modern text generation systems with better paraphrasing capabilities (Mathur et al., 2020). As a remedy, multiple higher-quality met- rics based on BERT and its extensions have been proposed in the last few years (Zhang et al., 2019; Zhao et al., 2019). In this work, we investigate whether these more recent metrics have better re- producibility properties, thus filling a gap for the newer paradigm of metrics. We have good …

Content truncated to 500 words