“Transforming” Delete, Retrieve, Generate Approach for Controlled Text Style Transfer

Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing, pages 3269–3279, Hong Kong, China, November 3–7, 2019. c⃝2019 Association for Computational Linguistics 3269 Transforming Delete, Retrieve, Generate Approach for Controlled Text Style Transfer Akhilesh Sudhakar, Bhargav Upadhyay, Arjun Maheswaran Agara {akhilesh, bhargav, arjun}@agaralabs.com Abstract Text style transfer is the task of transferring the style of text having certain stylistic attributes, while preserving non-stylistic or content infor- mation. In this work we introduce the Genera- tive Style Transformer (GST) - a new approach to rewriting sentences to a target style in the absence of parallel style corpora. GST lever- ages the power of both, large unsupervised pre-trained language models as well as the Transformer. GST is a part of a larger ‘Delete Retrieve Generate’ framework, in which we also propose a novel method of deleting style attributes from the source sentence by exploit- ing the inner workings of the Transformer. Our models outperform state-of-art systems across 5 datasets on sentiment, gender and political slant transfer. We also propose the use of the GLEU metric as an automatic metric of evalu- ation of style transfer, which we found to com- pare better with human ratings than the pre- dominantly used BLEU score. 1 Introduction Text style transfer is an important Natural Lan- guage Generation (NLG) task, and has wide- ranging applications from adapting conversational style in dialogue agents (Zhou et al., 2017), obfus- cating personal attributes (such as gender) to pre- vent privacy intrusion (Reddy and Knight, 2016), altering texts to be more formal or informal (Rao and Tetreault, 2018), to generating poetry (Yang et al., 2018). The main challenge faced in building style transfer systems is the lack of parallel cor- pora between sentences of a particular style and sentences of another, such that sentences in a pair differ only in style and not content (non-stylistic part of the sentence). This has given rise to meth- ods that circumvent the need for such parallel cor- pora. Previous approaches using non-parallel cor- pora, that employ learned latent representations to disentangle style and content from sentences, are typically adversarially trained (Hu et al., 2017; Shen et al., 2017; Fu et al., 2018). However these models a) are hard to train and take long to con- verge, b) need to be re-trained from scratch to change the trade-off between content retention and style transfer c) suffer from sparsity of latent dis- entangled representations, d) produce sentences of bad quality (according to human ratings) and e) do not offer fine-grained control over target style at- tributes. Li et al. (2018) find that style attributes are more often than not, localized to a small subset of words of a sentence. Building on this inductive bias, they model style transfer in a “Delete Retrieve Generate” framework (hereby referred to as DRG) which aims to 1) delete only the set of attribute words from a sentence to give the content, 2) re- trieve attribute words …

Content truncated to 500 words