Layer or Representation Space: What Makes BERT-based Evaluation Metrics Robust?

Proceedings of the 29th International Conference on Computational Linguistics, pages 3401–3411 October 12–17, 2022. 3401 Layer or Representation Space: What makes BERT-based Evaluation Metrics Robust? Doan Nam Long Vu1, Nafise Sadat Moosavi2, Steffen Eger3 1 Department of Computer Science, Technical University of Darmstadt, Germany 2 Department of Computer Science, The University of Sheffield, UK 3 NLLG, Faculty of Technology, Bielefeld University, Germany doannamlong.vu@stud.tu-darmstadt.de Abstract The evaluation of recent embedding-based eval- uation metrics for text generation is primarily based on measuring their correlation with hu- man evaluations on standard benchmarks. How- ever, these benchmarks are mostly from simi- lar domains to those used for pretraining word embeddings. This raises concerns about the (lack of) generalization of embedding-based metrics to new and noisy domains that con- tain a different vocabulary than the pretraining data. In this paper, we examine the robust- ness of BERTScore, one of the most popular embedding-based metrics for text generation. We show that (a) an embedding-based metric that has the highest correlation with human evaluations on a standard benchmark can have the lowest correlation if the amount of input noise or unknown tokens increases, (b) taking embeddings from the first layer of pretrained models improves the robustness of all metrics, and (c) the highest robustness is achieved when using character-level embeddings, instead of token-based embeddings, from the first layer of the pretrained model.1 1 Introduction Evaluating the quality of generated outputs by Nat- ural Language Generation (NLG) models is a chal- lenging and open problem. Human judgments can directly assess the quality of generated texts (Popovi´c, 2020; Escribe, 2019). However, human evaluation, either with experts or crowdsourcing, is expensive and time-consuming. Therefore, auto- matic evaluation metrics, which are fast and cheap, are commonly used alternatives for the rapid de- velopment of text generation systems (van der Lee et al., 2019). Traditional metrics such as BLEU (Papineni et al., 2002), METEOR (Banerjee and Lavie, 2005), and ROUGE (Lin, 2004) measure 1The code of our experiments is avail- able at https://github.com/long21wt/ robust-bert-based-metrics n-gram overlap between generated and reference texts. While these metrics are easy to use, they cannot correctly assess generated texts that contain novel words or a rephrasing of the reference text. Recent metrics like BERTScore (Zhang et al., 2020), MoverScore (Zhao et al., 2019), COMET (Rei et al., 2020), BARTScore (Yuan et al., 2021), and BLEURT (Sellam et al., 2020) adapt pretrained contextualized word embeddings to tackle this is- sue. These novel metrics have shown higher cor- relations with human judgments on various tasks and datasets (Ma et al., 2019; Mathur et al., 2020). However, the correlations are measured on stan- dard benchmarks containing text domains similar to those used for pretraining the embeddings them- selves. As a result, it is unclear how reliable these metrics are on domains and datasets containing words outside the vocabulary of the pretraining data. The goal of this paper is to investigate the robust- ness of embedding-based evaluation metrics on new and noisy domains that contain a higher ratio of unknown tokens …

Content truncated to 500 words