Contextual Text Style Transfer

Findings of the Association for Computational Linguistics: EMNLP 2020, pages 2915–2924 November 16 - 20, 2020. c⃝2020 Association for Computational Linguistics 2915 Contextual Text Style Transfer Yu Cheng1, Zhe Gan1, Yizhe Zhang2, Oussama Elachqar2, Dianqi Li3, Jingjing Liu1 1Microsoft Dynamics 365 AI Research 2Microsoft Research 3University of Washington {yu.cheng,zhe.gan,yizhe.zhang,ouelachq,jinjl}@microsoft.com, dianqili@uw.edu Abstract We introduce a new task, Contextual Text Style Transfer - translating a sentence into a desired style with its surrounding context taken into account. This brings two key chal- lenges to existing style transfer approaches: (i) how to preserve the semantic meaning of target sentence and its consistency with sur- rounding context during transfer; (ii) how to train a robust model with limited labeled data accompanied by context. To realize high-quality style transfer with natural con- text preservation, we propose a Context-Aware Style Transfer (CAST) model, which uses two separate encoders for each input sentence and its surrounding context. A classifier is fur- ther trained to ensure contextual consistency of the generated sentence. To compensate for the lack of parallel data, additional self- reconstruction and back-translation losses are introduced to leverage non-parallel data in a semi-supervised fashion. Two new bench- marks, Enron-Context and Reddit-Context, are introduced for formality and offensiveness style transfer. Experimental results on these datasets demonstrate the effectiveness of the proposed CAST model over state-of-the-art methods across style accuracy, content preser- vation and contextual consistency metrics.1 1 Introduction Text style transfer has been applied to many appli- cations (e.g., sentiment manipulation, formalized writing) with remarkable success. Early work relies on parallel corpora with a sequence-to-sequence learning framework (Bahdanau et al., 2015; Jham- tani et al., 2017). However, collecting parallel an- notations is highly time-consuming and expensive. There has also been studies on developing text style transfer models with non-parallel data (Hu et al., 1Code and datasets will be released at https:// github.com/ych133/CAST. 2017; Li et al., 2018; Prabhumoye et al., 2018; Subramanian et al., 2018), assuming that disentan- gling style information from semantic content can be achieved in an auto-encoding fashion with the introduction of additional regularizers (e.g., adver- sarial discriminators (Shen et al., 2017), language models (Yang et al., 2018)). Despite promising results, these techniques still have a long way to go for practical use. Most existing models focus on sentence-level rewriting. However, in real-world applications, sentences typ- ically reside in a surrounding paragraph context. In formalized writing, the rewritten span is expected to align well with the surrounding context to keep a coherent semantic flow. For example, to auto- matically replace a gender-biased sentence in a job description document, a style transfer model tak- ing the sentence out of context may not be able to understand the proper meaning of the statement and the intended message. Taking a single sen- tence as the sole input of a style transfer model may fail in preserving topical coherency between the generated sentence and its surrounding context, leading to low semantic and logical consistency on the paragraph level (see Example C in Table 4). Similar observations can be …

Content truncated to 500 words