On Learning Text Style Transfer with Direct Rewards

Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 4262–4273 June 6–11, 2021. ©2021 Association for Computational Linguistics 4262 On Learning Text Style Transfer with Direct Rewards Yixin Liu, Graham Neubig, John Wieting Carnegie Mellon University {yixinl2,gneubig,jwieting}@cs.cmu.edu Abstract In most cases, the lack of parallel corpora makes it impossible to directly train super- vised models for the text style transfer task. In this paper, we explore training algorithms that instead optimize reward functions that ex- plicitly consider different aspects of the style- transferred outputs. In particular, we leverage semantic similarity metrics originally used for fine-tuning neural machine translation models to explicitly assess the preservation of content between system outputs and input texts. We also investigate the potential weaknesses of the existing automatic metrics and propose effi- cient strategies of using these metrics for train- ing. The experimental results show that our model provides significant gains in both auto- matic and human evaluation over strong base- lines, indicating the effectiveness of our pro- posed methods and training strategies.1 1 Introduction Text style transfer aims to convert an input text into another generated text with a different style but the same basic semantics as the input. One major challenge in this setting is that many style transfer tasks lack parallel corpora, since the absence of hu- man references makes it impossible to train the text style transfer models using maximum likelihood estimation (MLE), which aims to maximize the predicted likelihood of the references. As a result, some of the earliest work (Shen et al., 2017; Hu et al., 2017; Fu et al., 2018) on unsupervised text style transfer proposed training algorithms that are still based on MLE by formulating the style transfer models as auto-encoders optimized with reconstruc- tion loss. Specifically, during training the model is tasked to generate a style-agnostic encoding and reconstruct the input text based on this encoding with style-specific embeddings or decoders. Dur- ing inference, the model aims to transfer the source 1Code and data are available at: https://github. com/yixinL7/Direct-Style-Transfer text style using the target style information. While these methods have seen empirical success, they face the inherent difficulty of coming up with a style-agnostic but content-preserving encoding – this is a non-trivial task and failure at this first step will diminish style transfer accuracy and content preservation of the final output. Another line of work (Xu et al., 2018; Pang and Gimpel, 2019; Luo et al., 2019) proposes train- ing algorithms based on rewards related to the au- tomatic evaluation metrics, which can assess the model performance more directly during training. This approach is conceptually similar to training algorithms that optimize models using rewards re- lated to the corresponding evaluation metrics for other NLP tasks, such as machine translation (Shen et al., 2016; Wieting et al., 2019a) or text summa- rization (Paulus et al., 2018; Li et al., 2019). As for unsupervised style transfer, the widely used au- tomatic metrics mainly attend to three desiderata: (1) …

Content truncated to 500 words