NAST: A Non-Autoregressive Generator with Word Alignment for Unsupervised Text Style Transfer

Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021, pages 1577–1590 August 1–6, 2021. ©2021 Association for Computational Linguistics 1577 NAST: A Non-Autoregressive Generator with Word Alignment for Unsupervised Text Style Transfer Fei Huang, Zikai Chen, Chen Henry Wu, Qihan Guo, Xiaoyan Zhu, Minlie Huang∗ The CoAI group, DCST; Institute for Artificial Intelligence; State Key Lab of Intelligent Technology and Systems; Beijing National Research Center for Information Science and Technology; Tsinghua University, Beijing 100084, China. f-huang18@mails.tsinghua.edu.cn natnstart@gmail.com henrychenwu98@gmail.com gqh18@mails.tsinghua.edu.cn zxy-dcs@tsinghua.edu.cn aihuang@tsinghua.edu.cn Abstract Autoregressive models have been widely used in unsupervised text style transfer. Despite their success, these models still suffer from the content preservation problem that they usu- ally ignore part of the source sentence and generate some irrelevant words with strong styles. In this paper, we propose a Non- Autoregressive generator for unsupervised text Style Transfer (NAST), which alleviates the problem from two aspects. First, we observe that most words in the transferred sentence can be aligned with related words in the source sen- tence, so we explicitly model word alignments to suppress irrelevant words. Second, existing models trained with the cycle loss align sen- tences in two stylistic text spaces, which lacks fine-grained control at the word level. The pro- posed non-autoregressive generator focuses on the connections between aligned words, which learns the word-level transfer between styles. For experiments, we integrate the proposed generator into two base models and evaluate them on two style transfer tasks. The re- sults show that NAST can significantly im- prove the overall performance and provide ex- plainable word alignments. Moreover, the non- autoregressive generator achieves over 10x speedups at inference. Our codes are available at https://github.com/thu-coai/NAST. 1 Introduction Text style transfer aims at changing the text style while preserving the style-irrelevant contents, which has a wide range of applications, e.g., senti- ment transfer (Shen et al., 2017), text formalization (Rao and Tetreault, 2018), and author imitation (Jhamtani et al., 2017). Due to the lack of parallel training data, most works focus on unsupervised text style transfer using non-parallel stylistic data. The cycle consistency loss (Zhu et al., 2017), a.k.a. the back-translation loss (Lample et al., 2018, *Corresponding author: Minlie Huang. Not great, but good atmosphere and great service Not terrible, but not very good Source: Transferred: Autoregressive Generation (a) Existing Style Transfer Model Not perfect , but indeed very good Not terrible , but not very good Source: Target: (b) Observation of Word Alignment Not perfect , but very good indeed Not terrible , but not very good very Not terrible , but good [Mask] Source: Transferred: Step 1. Alignment Prediction Aligned: (c) NAST (Ours) Two Step Decomposition Step 2. Non-autoregressive Generation 6 1 2 3 4 7 0 5 1 2 3 4 6 7 Figure 1: Sentiment transfer examples (negative to pos- itive). (a) Existing models without word alignments may generate words irrelevant to the source sen- tence. (b) An example of word alignments between the source and target sentences. Arrows connect aligned words (identical or relevant), and blue words …

Content truncated to 500 words