TextSETTR: Few-Shot Text Style Extraction and Tunable Targeted Restyling

Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing, pages 3786–3800 August 1–6, 2021. ©2021 Association for Computational Linguistics 3786 TextSETTR: Few-Shot Text Style Extraction and Tunable Targeted Restyling Parker Rileya∗, Noah Constantb, Mandy Guob, Girish Kumarc∗, David Uthusb, Zarana Parekhb aUniversity of Rochester bGoogle Research cStanford University Abstract We present a novel approach to the problem of text style transfer. Unlike previous ap- proaches requiring style-labeled training data, our method makes use of readily-available un- labeled text by relying on the implicit connec- tion in style between adjacent sentences, and uses labeled data only at inference time. We adapt T5 (Raffel et al., 2020), a strong pre- trained text-to-text model, to extract a style vector from text and use it to condition the de- coder to perform style transfer. As our label- free training results in a style vector space en- coding many facets of style, we recast trans- fers as “targeted restyling” vector operations that adjust specific attributes of the input while preserving others. We demonstrate that train- ing on unlabeled Amazon reviews data results in a model that is competitive on sentiment transfer, even compared to models trained fully on labeled data. Furthermore, applying our novel method to a diverse corpus of unla- beled web text results in a single model capa- ble of transferring along multiple dimensions of style (dialect, emotiveness, formality, polite- ness, sentiment) despite no additional training and using only a handful of exemplars at infer- ence time. 1 Introduction There has been a recent surge of interest in text style transfer, with the aim of training models able to modify specific attributes of input text (e.g., sen- timent or formality) while preserving the remaining content. For example, a sentiment transfer model might transform the input “best book ever!” into “worst book ever!”, while a formality transfer model might change the same input into “This is the best book I have ever read.” In these contexts, we de- fine “style” as the attributes intended to be changed, ∗ Work done while at Google Research. Please di- rect correspondence to priley3@cs.rochester.edu, nconstant@google.com and xyguo@google.com. while “content” consists of the attributes intended to be preserved.1 Work in this area falls into three categories. Su- pervised approaches like Jhamtani et al. (2017) transfer between pre-selected styles, and rely on parallel training data to learn the desired in- put/output correspondence. This method is limited by the availability of parallel corpora. So-called “unsupervised” approaches like Li et al. (2018) and Lample et al. (2019) remove the need for par- allel data, but still require that all training exam- ples have style labels, and are limited to transfer between a pre-specified set of styles. Few-shot approaches like that of Xu et al. (2020) remove the need for any training labels, instead using a small number of labeled examples during infer- ence. While the most challenging, this offers the potential for transferring between arbitrary styles at inference time …

Content truncated to 500 words