Automatic Text Evaluation through the Lens of Wasserstein Barycenters

Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 10450–10466 November 7–11, 2021. c⃝2021 Association for Computational Linguistics 10450 Automatic Text Evaluation through the Lens of Wasserstein Barycenters Pierre Colombo1,2, Guillaume Staerman1, Chloe Clavel1, Pablo Piantanida ∗ 1LTCI, Telecom Paris, Institut Polytechnique de Paris, 2IBM GBS France, ∗CentraleSupelec-CNRS, Universite Paris-Saclay 1firstname.lastname@telecom-paris.fr, 2pierre.colombo@ibm.com, ∗pablo.piantanida@centralesupelec.fr, Abstract A new metric BaryScore to evaluate text generation based on deep contextualized em- beddings (e.g., BERT, Roberta, ELMo) is in- troduced. This metric is motivated by a new framework relying on optimal transport tools, i.e., Wasserstein distance and barycenter. By modelling the layer output of deep contextu- alized embeddings as a probability distribu- tion rather than by a vector embedding; this framework provides a natural way to aggregate the different outputs through the Wasserstein space topology. In addition, it provides the- oretical grounds to our metric and offers an alternative to available solutions (e.g., Mover- Score and BertScore). Numerical evaluation is performed on four different tasks: machine translation, summarization, data2text genera- tion and image captioning. Our results show that BaryScore outperforms other BERT based metrics and exhibits more consistent be- haviour in particular for text summarization. 1 Introduction Automatic Evaluation (AE) of Natural Language Generation (NLG) is a key problem towards better systems (Specia et al., 2010). It allows to assess the quality of generated text without relying on hu- man evaluation campaigns that are expensive and time consuming (Belz and Reiter, 2006; Sai et al., 2020). For instance, it becomes crucial to design automatic and effective metrics with simultaneous goals: (i) to be able to compare, to control and to debug systems without relying on human anno- tators (Peyrard, 2019a,b); and (ii) to improve the learning phase of models by deriving losses that are better surrogate of human judgment than the widely used cross-entropy loss (Clark et al., 2019). A plethora of automatic metrics has been intro- duced these last few years and may be grouped into two general classes: trained (Ma et al., 2017; Shi- manaka et al., 2018; Lowe et al., 2016; Lita et al., 2005) and untrained metrics (Doddington, 2002; Popovi´c, 2015). In this paper, we mainly focus on untrained metrics that can be further split into three subgroups: string matching (Papineni et al., 2002; Lin, 2004; Banerjee and Lavie, 2005; Doddington, 2002; Popovi´c, 2015), edit based (Leusch et al., 2006; Snover et al., 2006; Wang et al., 2016) and embedding based metrics (Chow et al., 2019; Kus- ner et al., 2015; Lo and Wu, 2011; Lo, 2019). Both string matching and edit based metrics fail to assign reliable scores when reference and candidate con- vey the same meaning with distinct surface forms (Reiter and Belz, 2009) (e.g., case of synonyms and paraphrases). These shortcomings have been addressed by metrics based on continuous represen- tations. Recently, they have benefited from contex- tual embeddings such as ELMO (Peters et al., 2018) and BERT (Devlin et al., 2018; Liu et al., 2019b). Perhaps, the most known are BERTScore (Zhang et …

Content truncated to 500 words