Dialect-robust Evaluation of Generated Text

Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics Volume 1: Long Papers, pages 6010–6028 July 9-14, 2023 ©2023 Association for Computational Linguistics Dialect-robust Evaluation of Generated Text Jiao Sun1,2∗ Thibault Sellam1 Elizabeth Clark1 Tu Vu1,3∗ Timothy Dozat1 Dan Garrette1 Aditya Siddhant1 Jacob Eisenstein1 Sebastian Gehrmann1 1Google Deepmind 2University of Southern California 3University of Massachusetts Amherst nano-dialect-eval@google.com Abstract Text generation metrics that are not robust to dialect variation make it impossible to tell how well systems perform for many groups of users, and can even penalize systems for producing text in lower-resource dialects. In this paper, we introduce a suite of methods to assess whether metrics are dialect robust. These methods show that state-of-the-art metrics are not dialect ro- bust: they often prioritize dialect similarity over semantics, preferring outputs that are seman- tically incorrect over outputs that match the semantics of the reference but contain dialect differences. As a step towards dialect-robust metrics for text generation, we propose NANO, which introduces regional and language infor- mation to the metric’s pretraining. NANO significantly improves dialect robustness while preserving the correlation between automated metrics and human ratings. It also enables a more ambitious approach to evaluation, dialect awareness, in which system outputs are scored by both semantic match to the reference and appropriateness in any specified dialect. 1 Introduction Most natural language generation (NLG) evalua- tion metrics compare a system output against a human-written reference. References are usually drawn from a relatively narrow range of linguis- tic styles. They often exclude varieties like Indian English or Iberian Portuguese, which are geograph- ical dialects with millions of speakers. As a result, outputs in dialects that are not represented in the reference may score poorly, discouraging the de- velopment of systems to meet the needs of these language communities. Although contemporary metrics such as COMET (Rei et al., 2020) can be reference-free, they still rely on training data and rater pools that do not cover all dialects of interest, leading to a high number of out-of-domain dialects. ∗Work done while Jiao and Tu interning at Google. The performance of evaluation metrics on these out-of-domain dialects has not been quantified. We define a dialect-robust evaluation metric as one that produces the same score for system outputs that share the same semantics, but are expressed in different dialects. To understand whether cur- rent evaluation metrics are dialect-robust, we pro- pose to quantify the dialect robustness at the dialect feature-level and sentence-level. The analyses mea- sure the dialect-sensitivity of evaluation metrics by comparing semantics-preserving dialect edits to perturbations that change the meaning of sentences. Through our analyses, we demonstrate that mul- tiple state-of-the-art NLG evaluation metrics are not robust to dialects of Mandarin, English, and Portuguese. In many cases, system outputs that are perturbed so as to differ semantically from the ref- erence score higher than outputs in which the only change is to the dialect. With the goal of increas- ing the dialect robustness and without performance degradation on standard benchmarks, we propose …

Content truncated to 500 words