Text Style Transferring via Adversarial Masking and Styled Filling

Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 7654 - 7663 December 7-11, 2022 ©2022 Association for Computational Linguistics Text Style Transferring via Adversarial Masking and Styled Filling Jiarui Wang1, Richong Zhang1, 2 ∗, Junfan Chen1, Jaein Kim1, Yongyi Mao3 1SKLSDE, Beihang University, Beijing, China 2Zhongguancun Laboratory, Beijing, China 3 School of Electrical Engineering and Computer Science, University of Ottawa, Canada {wangjr,zhangrc,chenjf}@act.buaa.edu.cn jaein@buaa.edu.cn,ymao@uottawa.ca Abstract Text style transfer is an important task in nat- ural language processing with broad applica- tions. Existing models following the masking and filling scheme suffer two challenges: the word masking procedure may mistakenly re- move unexpected words and the selected words in the word filling procedure may lack diversity and semantic consistency. To tackle both chal- lenges, in this study, we propose a style transfer model, with an adversarial masking approach and a styled filling technique (AMSF). Specif- ically, AMSF first trains a mask predictor by adversarial training without manual configura- tion. Then two additional losses, i.e. an entropy maximization loss and a consistency regular- ization loss, are introduced in training the word filling module to guarantee the diversity and semantic consistency of the transferred texts. Experimental results and analysis on two bench- mark text style transfer data sets demonstrate the effectiveness of the proposed approaches. 1 Introduction Stylistic attributes of natural language text have in- trigued researchers in natural language processing (NLP) for a long time. Text style transfer aims to convert the text into a target style while preserv- ing the content of the source text, which has broad industrial applications such as computer-aided writ- ing (Klahold and Fathi, 2020) and advertising sys- tems (Jin et al., 2020). Due to the lack of parallel data, text style transfer is usually treated as an un- supervised learning task. One of the mainstream approaches is to exploit the Autoencoder to disentangle the style and con- tent representation of the source text, then use the decoder to generate a transferred text with the disentangled content representation and the target style (Fu et al., 2018; John et al., 2018; Shen et al., 2017; Nangi et al., 2021). The weakness of such ∗Corresponding author a sequence-to-sequence manner is that the model usually performs poorly on content preservation because the intrinsic of generative models bring a significant difference between source and target text. Another branch of models follows the mask- ing and filling scheme (Devlin et al., 2018; Xu et al., 2018; Sudhakar et al., 2019; Wu et al., 2019), which first explicitly masks out the stylistic words of the source style in the text and then replaces them with words from the target style. These models usually use either average attention scores (Bahdanau et al., 2014) or the higher frequency ratio as criteria to select the stylistic words in the word masking pro- cedure. However, these masking approaches may mistakenly remove the non-stylistic words when the attention distribution is insignificant (Lee et al., 2021). When filling the masked words, previous works utilize BERT (Devlin …

Content truncated to 500 words