Evaluating the Evaluation Metrics for Style Transfer: A Case Study in Multilingual Formality Transfer

Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 1321–1336 November 7–11, 2021. c⃝2021 Association for Computational Linguistics 1321 Evaluating the Evaluation Metrics for Style Transfer: A Case Study in Multilingual Formality Transfer Eleftheria Briakou , Sweta Agrawal , Joel Tetreault , Marine Carpuat University of Maryland, Dataminr, Inc. ebriakou@cs.umd.edu, sweagraw@cs.umd.edu, jtetreault@dataminr.com, marine@cs.umd.edu Abstract While the field of style transfer (ST) has been growing rapidly, it has been hampered by a lack of standardized practices for automatic evaluation. In this paper, we evaluate lead- ing ST automatic metrics on the oft-researched task of formality style transfer. Unlike previ- ous evaluations, which focus solely on English, we expand our focus to Brazilian-Portuguese, French, and Italian, making this work the first multilingual evaluation of metrics in ST. We outline best practices for automatic evaluation in (formality) style transfer and identify sev- eral models that correlate well with human judgments and are robust across languages. We hope that this work will help accelerate de- velopment in ST, where human evaluation is often challenging to collect. 1 Introduction Textual style transfer (ST) is defined as a generation task where a text sequence is paraphrased while controlling one aspect of its style (Jin et al., 2020). For instance, the informal sentence in Italian “in bocca al lupo!” (i.e., “good luck”) is rewritten to the formal version “Ti rivolgo un sincero augurio!” (i.e., “I send you a sincere wish!”). Despite the growing attention on ST in the NLP literature (Jin et al., 2020), progress is hampered by a lack of stan- dardized and reliable automatic evaluation metrics. Standardizing the latter would allow for quicker de- velopment of new methods and comparison to prior art without relying on time and cost-intensive hu- man evaluation that is currently employed by more than 70% of ST papers (Briakou et al., 2021a). ST is usually evaluated across three dimensions: style transfer (i.e., has the style of the generated out- put changed as intended?), meaning preservation (i.e., are the semantics of the input preserved?), and fluency (i.e., is the output well-formed?). As we will see, a wide range of automatic evaluation met- rics and models has been used to quantify each of these dimensions. For example, prior work has em- ployed as many as nine different automatic systems to rate formality alone (see Table 1). However, it is not clear how different automatic metrics compare to each other and how well they agree with human judgments. Furthermore, previous studies of auto- matic evaluation have exclusively focused on the English language (Yamshchikov et al., 2021; Pang, 2019; Pang and Gimpel, 2019; Tikhonov et al., 2019; Mir et al., 2019); yet, ST requires evaluation methods that generalize reliably beyond English. We address these limitations by conducting a controlled empirical comparison of commonly used automatic evaluation metrics. Concretely, for all three evaluation dimensions, we compile a list of different automatic evaluation approaches used in prior ST work and study how well they correlate with human judgments. We choose to build …

Content truncated to 500 words