Exploring Non-Autoregressive Text Style Transfer

Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 9267–9278 November 7–11, 2021. c⃝2021 Association for Computational Linguistics 9267 Exploring Non-Autoregressive Text Style Transfer Yun Ma Department of Computing The Hong Kong Polytechnic University mayun371@gmail.com Qing Li Department of Computing The Hong Kong Polytechnic University csqli@comp.polyu.edu.hk Abstract In this paper, we explore Non-AutoRegressive (NAR) decoding for unsupervised text style transfer. We first propose a base NAR model by directly adapting the common training scheme from its AutoRegressive (AR) counterpart. De- spite the faster inference speed over the AR model, this NAR model sacrifices its transfer performance due to the lack of conditional de- pendence between output tokens. To this end, we investigate three techniques, i.e., knowledge distillation, contrastive learning, and iterative decoding, for performance enhancement. Ex- perimental results on two benchmark datasets suggest that, although the base NAR model is generally inferior to AR decoding, their per- formance gap can be clearly narrowed when empowering NAR decoding with knowledge distillation, contrastive learning, and iterative decoding. 1 Introduction Text Style Transfer (TST) aims at altering a stylis- tic attribute (e.g., sentiment) of the given text to a target value, without changing the style-agnostic semantics. Due to the difficulty in collecting par- allel training corpus, most exiting methods (Shen et al., 2017; Xu et al., 2018; Luo et al., 2019; Zhou et al., 2020) address the task in an unsupervised setting. Through techniques such as auto-encoding, back-translation, or adversarial learning, the task is converted to self-supervised problems and great empirical progress has been made. However, cur- rent methods employ autoregressive (AR) decoding which generates each output token conditioned on the previously generated ones, leading to low par- allelizability and high latency for pragmatic use. Recently, non-autoregressive (NAR) decoding has attracted much attention in neural machine translation (NMT) (Gu et al., 2018). NAR decod- ing eliminates the conditional dependencies among the output tokens and generates them in parallel, thus reducing the decoding time-complexity from O(T) to O(1) for outputs with length T. Without modeling the dependency, the advantage on decod- ing speed comes at the cost of reduced performance. To address this issue, knowledge distillation is em- ployed to transfer the knowledge from AR models to NAR models (Gu et al., 2018). Furthermore, existing works resort to various regularization tech- niques (Wang et al., 2019) to constrain the output or Semi-AutoRegressive (SAR) decoding (Wang et al., 2018; Ghazvininejad et al., 2019) as a speed- performance tradeoff. In this paper, we explore NAR decoding for unsu- pervised TST to enable faster inference with better parallelism. To the best of our knowledge, this is the first work to study NAR models for TST. Firstly, a base NAR model is proposed by directly adapting the widely used training objectives from AR mod- els. As with NMT, the base NAR model underper- forms the AR model. To narrow their performance gap, we propose to enhance NAR decoding from three perspectives: the data perspective by knowl- edge distillation, the regularization perspective by contrastive learning, …

Content truncated to 500 words