SESCORE2: Learning Text Generation Evaluation via Synthesizing Realistic Mistakes

Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics Volume 1: Long Papers, pages 5166–5183 July 9-14, 2023 ©2023 Association for Computational Linguistics SESCORE2: Learning Text Generation Evaluation via Synthesizing Realistic Mistakes Wenda Xu¶, Xian Qian†, Mingxuan Wang†, Lei Li¶, William Yang Wang¶ †ByteDance, ¶UC Santa Barbara {wendaxu, leili, william}@cs.ucsb.edu {wangmingxuan.89, qian.xian}@bytedance.com Abstract Is it possible to train a general metric for eval- uating text generation quality without human- annotated ratings? Existing learned metrics either perform unsatisfactorily across text gen- eration tasks or require human ratings for train- ing on specific tasks. In this paper, we pro- pose SESCORE2, a self-supervised approach for training a model-based metric for text gen- eration evaluation. The key concept is to syn- thesize realistic model mistakes by perturb- ing sentences retrieved from a corpus. The primary advantage of the SESCORE2 is its ease of extension to many other languages while providing reliable severity estimation. We evaluate SESCORE2 and previous methods on four text generation tasks across three lan- guages. SESCORE2 outperforms unsupervised metric PRISM on four text generation evalua- tion benchmarks, with a Kendall improvement of 0.078. Surprisingly, SESCORE2 even out- performs the supervised BLEURT and COMET on multiple text generation tasks. The code and data are available at https://github.com/ xu1998hz/SEScore21. 1 Introduction Recently, researchers made significant progress in text generation: translation (Birch, 2021), struc- tured data-to-text (Gardent et al., 2017), dialogue generation (Vinyals and Le, 2015), and summariza- tion (Chopra et al., 2016). Automatic metrics are essential for the development of text generation models as they replace expensive human labor and are able to evaluate the generation performance (Celikyilmaz et al., 2020), as well as guide the gen- eration process (Unanue et al., 2021; Freitag et al., 2022). How can we efficiently and effectively train a metric for general text generation tasks? Depending on the inputs, we can categorize eval- uation metrics into source-based, hybrid-based, and 1Part of the work is done while WX is an intern at ByteDance. reference-based metrics. Source-based metrics es- timate text quality through the source and are use- ful when reference is noisy or unavailable (Louis and Nenkova, 2013; Kepler et al., 2019), but they may produce sub-optimal results and explore spuri- ous correlations (Durmus et al., 2022). Reference- based metrics, when paired with high-quality refer- ences, can reflect text generation quality, regardless of source modalities (e.g audio and triples). Hybrid metric COMET (Rei et al., 2020) uses both source and reference. In this work, we aim to construct a reference-based metric, as it is invariant to the source modality, making it suitable for use across various tasks. Although learned metrics have been shown to be more effective than rule-based metrics (e.g BLEU (Papineni et al., 2002)), they still have limitations in terms of evaluation capability and applicabil- ity to specific tasks. Supervised metrics such as BLEURT (Sellam et al., 2020) and COMET (Rei et al., 2020) are superior in evaluation, but re- stricted to tasks with human ratings of generated text. Unsupervised metrics, such …

Content truncated to 500 words