Style Transfer for Texts: Retrain, Report Errors, Compare with Rewrites

Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing, pages 3936–3945, Hong Kong, China, November 3–7, 2019. c⃝2019 Association for Computational Linguistics 3936 Style Transfer for Texts: Retrain, Report Errors, Compare with Rewrites Alexey Tikhonov∗ Yandex Berlin, Germany altsoph@gmail.com Viacheslav Shibaev∗ Ural Federal University Ekaterinburg, Russia Aleksander Nagaev Ural Federal University/Sberbank Ekaterinburg, Russia Aigul Nugmanova Speech Technology Center St. Petersburg, Russia Ivan P. Yamshchikov∗ Max Planck Institute for Mathematics in the Sciences Leipzig, Germany ivan@yamshchikov.info Abstract This paper shows that standard assessment methodology for style transfer has several sig- nificant problems. First, the standard metrics for style accuracy and semantics preservation vary significantly on different re-runs. There- fore one has to report error margins for the obtained results. Second, starting with cer- tain values of bilingual evaluation understudy (BLEU) between input and output and accu- racy of the sentiment transfer the optimization of these two standard metrics diverge from the intuitive goal of the style transfer task. Finally, due to the nature of the task itself, there is a specific dependence between these two met- rics that could be easily manipulated. Under these circumstances, we suggest taking BLEU between input and human-written reformula- tions into consideration for benchmarks. We also propose three new architectures that out- perform state of the art in terms of this metric. 1 Introduction Deep generative models attract a lot of attention in recent years (Hu et al., 2017b). Such methods as variational autoencoders (Kingma and Welling, 2013) or generative adversarial networks (Good- fellow et al., 2014) are successfully applied to a variety of machine vision problems including im- age generation (Radford et al., 2017), learning interpretable image representations (Chen et al., 2016) and style transfer for images (Gatys et al., 2016). However, natural language generation is more challenging due to many reasons, such as the discrete nature of textual information (Hu et al., 2017a), the absence of local information continu- ity and non-smooth disentangled representations (Bowman et al., 2015). Due to these difficulties, text generation is mostly limited to specific narrow ∗Equal contribution applications and is usually working in supervised settings. Content and style are deeply fused in natural lan- guage, but style transfer for texts is often addressed in the context of disentangled latent representations (Hu et al., 2017a; Shen et al., 2017; Fu et al., 2018; John et al., 2018; Romanov et al., 2018; Tian et al., 2018). Intuitive understanding of this problem is apparent: if an input text has some attribute A, a system generates new text similar to the input on a given set of attributes with only one attribute A changed to the target attribute ˜A. In the major- ity of previous works, style transfer is obtained through an encoder-decoder architecture with one or multiple style discriminators to learn disentan- gled representations. The encoder takes a sentence as an input and generates a style-independent con- tent representation. The decoder then takes the content representation and the target …

Content truncated to 500 words