Semi-Supervised Formality Style Transfer with Consistency Training

Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics Volume 1: Long Papers, pages 4689 - 4701 May 22-27, 2022 c⃝2022 Association for Computational Linguistics Semi-Supervised Formality Style Transfer with Consistency Training Ao Liu, An Wang, Naoaki Okazaki Tokyo Institute of Technology liu.ao@nlp.c.titech.ac.jp wang@de.cs.titech.ac.jp okazaki@c.titech.ac.jp Abstract Formality style transfer (FST) is a task that in- volves paraphrasing an informal sentence into a formal one without altering its meaning. To address the data-scarcity problem of existing parallel datasets, previous studies tend to adopt a cycle-reconstruction scheme to utilize addi- tional unlabeled data, where the FST model mainly benefits from target-side unlabeled sen- tences. In this work, we propose a simple yet effective semi-supervised framework to better utilize source-side unlabeled sentences based on consistency training. Specifically, our ap- proach augments pseudo-parallel data obtained from a source-side informal sentence by en- forcing the model to generate similar outputs for its perturbed version. Moreover, we em- pirically examined the effects of various data perturbation methods and propose effective data filtering strategies to improve our frame- work. Experimental results on the GYAFC benchmark demonstrate that our approach can achieve state-of-the-art results, even with less than 40% of the parallel data1. 1 Introduction Formality style transfer (FST) (Rao and Tetreault, 2018) has garnered growing attention in the text style transfer community, which aims to transform an informal-style sentence into a formal one while preserving its meaning. The large amount of user- generated data from online resources like tweets often contain informal expressions such as slang words (e.g., gonna), wrong capitalization or punc- tuations, and grammatical or spelling errors. FST can clean and formalize such noisy data, to benefit downstream NLP applications such as sentiment classification (Yao and Yu, 2021a). Some examples of FST data are presented in Table 1. With the release of the FST benchmark Gram- marly Yahoo Answers Corpus (GYAFC) (Rao and 1Code available at https://github.com/Aolius/ semi-fst. Informal TITANIC I THINK IT COST ABOUT 300 MILLION Formal I think that Titanic cost around 300 million dollars. Informal being condiderate of her feelings and needs Formal I am being considerate of her personal needs and feelings. Table 1: Examples of informal-formal sentence pairs. Tetreault, 2018), previous studies on FST tend to employ neural networks such as sequence-to- sequence (seq2seq) models to utilize parallel (infor- mal and formal) sentence pairs. However, GYAFC only contains 100k parallel examples, which lim- its the performance of neural network models. Several approaches have been developed to ad- dress the data-scarcity problem by utilizing un- labeled sentences. In a previous study, Zhang et al. (2020) proposed several effective data aug- mentations methods, such as back-translation, to augment parallel data. Another line of research (Shang et al., 2019; Xu et al., 2019; Chawla and Yang, 2020) conducted semi-supervised learning (SSL) in a cycle-reconstruction manner, where both forward and backward transfer models were jointly trained while benefiting each other by gen- erating pseudo-parallel data from unlabeled sen- tences. Under this setting, both additional informal and formal sentences are utilized; however, …

Content truncated to 500 words