Unsupervised Text Style Transfer with Padded Masked Language Models

Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing, pages 8671–8680, November 16–20, 2020. c⃝2020 Association for Computational Linguistics 8671 Unsupervised Text Style Transfer with Padded Masked Language Models Eric Malmi Google Research emalmi@google.com Aliaksei Severyn Google Research severyn@google.com Sascha Rothe Google Research rothe@google.com Abstract We propose MASKER, an unsupervised text- editing method for style transfer. To tackle cases when no parallel source–target pairs are available, we train masked language models (MLMs) for both the source and the target domain. Then we find the text spans where the two models disagree the most in terms of likelihood. This allows us to identify the source tokens to delete to transform the source text to match the style of the target domain. The deleted tokens are replaced with the tar- get MLM, and by using a padded MLM vari- ant, we avoid having to predetermine the num- ber of inserted tokens. Our experiments on sentence fusion and sentiment transfer demon- strate that MASKER performs competitively in a fully unsupervised setting. Moreover, in low- resource settings, it improves supervised meth- ods’ accuracy by over 10 percentage points when pre-training them on silver training data generated by MASKER. 1 Introduction Text-editing methods (Dong et al., 2019; Malmi et al., 2019; Awasthi et al., 2019; Mallinson et al., 2020), that target monolingual sequence transduc- tion tasks like sentence fusion, grammar correction, and text simplification, are typically more data- efficient than the traditional sequence-to-sequence methods, but they still require substantial amounts of parallel training examples to work well. When parallel source–target training pairs are difficult to obtain, it is often still possible to collect non- parallel examples for the source and the target do- main separately. For instance, negative and positive reviews can easily be collected based on the numer- ical review scores associated with them, which has led to a large body of work on unsupervised text style transfer, e.g., (Yang et al., 2018; Shen et al., 2017; Wu et al., 2019; Li et al., 2018). The existing unsupervised style transfer methods aim at transforming a source text so that its style matches the target domain but its content stays otherwise unaltered. This is commonly achieved via text-editing performed in two steps: using one model to identify the tokens to delete and another model to infill the deleted text slots (Li et al., 2018; Xu et al., 2018; Wu et al., 2019). In contrast, we propose a more unified approach, showing that both of these steps can be completed using a single model, namely a masked language model (MLM) (Devlin et al., 2019). MLM is a natural choice for infilling the deleted text spans, but we can also use it to identify the tokens to delete by finding the spans where MLMs trained on the source and the target domain disagree in terms of likelihood. This is inspired by the recent observation that MLMs are effective at estimating (pseudo) likelihoods of texts (Wang and Cho, 2019; Salazar et al., 2020). Moreover, by …

Content truncated to 500 words