Evaluating Style Transfer for Text

Proceedings of NAACL-HLT 2019, pages 495–504 Minneapolis, Minnesota, June 2 - June 7, 2019. c⃝2019 Association for Computational Linguistics 495 Evaluating Style Transfer for Text Remi Mir1, Bjarke Felbo2, Nick Obradovich2, Iyad Rahwan2 1Department of EECS, Massachusetts Institute of Technology 2Media Lab, Massachusetts Institute of Technology {rmir, bfelbo, nobradov, irahwan}@mit.edu Abstract Research in the area of style transfer for text is currently bottlenecked by a lack of stan- dard evaluation practices. This paper aims to alleviate this issue by experimentally iden- tifying best practices with a Yelp sentiment dataset. We specify three aspects of interest (style transfer intensity, content preservation, and naturalness) and show how to obtain more reliable measures of them from human eval- uation than in previous work. We propose a set of metrics for automated evaluation and demonstrate that they are more strongly corre- lated and in agreement with human judgment: direction-corrected Earth Mover’s Distance, Word Mover’s Distance on style-masked texts, and adversarial classification for the respective aspects. We also show that the three examined models exhibit tradeoffs between aspects of in- terest, demonstrating the importance of evalu- ating style transfer models at specific points of their tradeoff plots. We release software with our evaluation metrics to facilitate research. 1 Introduction Style transfer in text is the task of changing an attribute (style) of an input, while retaining non- attribute related content (referred to simply as con- tent for brevity in this paper).1 For instance, pre- vious work has modified text to make it more pos- itive (Shen et al., 2017), romantic (Li et al., 2018), or politically slanted (Prabhumoye et al., 2018). Some style transfer models enable modifica- tions by manipulating latent representations of the text (Shen et al., 2017; Zhao et al., 2018; Fu et al., 2018), while others identify and replace style- related words directly (Li et al., 2018). Regardless of approach, they are hard to compare as there is 1This definition of style transfer makes a simplifying as- sumption that “style” words can be distinguished from “con- tent” words, or words carrying relatively less or no stylistic weight, such as “caf`e” in “What a nice caf`e.” The definition is motivated by penalizing unnecessary changes to content words, e.g. “What a nice caf`e” to “This is an awful caf`e.” currently neither a standard set of evaluation prac- tices, nor a clear definition of which exact aspects to evaluate. In Section 2, we define three key as- pects to consider. In Section 3, we summarize is- sues with previously used metrics. Many rely on human ratings, which can be expensive and time- consuming to obtain. To address these issues, in Section 4, we con- sider how to obtain more reliable measures of human judgment for aspects of interest, and au- tomated methods more strongly correlated with human judgment than previously used methods. Lastly, in Section 5, we show that the three ex- amined models exhibit aspect tradeoffs, highlight- ing the importance of evaluating style transfer models at specific points of their tradeoff plots. We release …

Content truncated to 500 words