Formality Style Transfer with Shared Latent Space

Proceedings of the 28th International Conference on Computational Linguistics, pages 2236–2249 Barcelona, Spain (Online), December 8-13, 2020 2236 Formality Style Transfer with Shared Latent Space Yunli Wang†, Yu Wu♦, Lili Mou‡, Zhoujun Li† , Wenhan Chao† †State Key Lab of Software Development Environment, Beihang University, Beijing, China ♦Microsoft Research, Beijing, China ‡Dept. Computing Science, University of Alberta, Edmonton, Canada Alberta Machine Intelligence Institute (Amii) {wangyunli,lizj,chaowenhan}@buaa.edu.cn Wu.Yu@microsoft.com doublepower.mou@gmail.com Abstract Conventional approaches for formality style transfer borrow models from neural machine trans- lation, which typically requires massive parallel data for training. However, the dataset for for- mality style transfer is considerably smaller than translation corpora. Moreover, we observe that informal and formal sentences closely resemble each other, which is different from the translation task where two languages have different vocabularies and grammars. In this paper, we present a new approach, Sequence-to-Sequence with Shared Latent Space (S2S-SLS), for formality style transfer, where we propose two auxiliary losses and adopt joint training of bi-directional transfer and auto-encoding. Experimental results show that S2S-SLS (with either RNN or Transformer architectures) consistently outperforms baselines in various settings, especially when we have limited data.1 1 Introduction The formality analysis of text plays an important role in natural language processing (NLP)-related ap- plications, such as style analysis and human-computer interaction (Pavlick and Tetreault, 2016). With the advance of neural networks for sentence generation, it is possible to synthesize a new sentence, changing the formality style of an input sentence while retaining its meaning. This is referred to as formality style transfer in Rao and Tetreault (2018). Rao and Tetreault (2018) in the meantime constructed the GYAFC2 dataset, where they make crowd- sourcing efforts to manually write 119K formal sentences from informal ones (examples shown in Ta- ble 1). It fertilizes the research of formality style transfer with parallel data, as most other style-transfer datasets (e.g., sentiment) are non-parallel, with only a style label available for each sentence. In previous work, researchers have applied machine translation frameworks to style transfer when par- allel data are available. For example, Xu et al. (2012) adopt phrase-based statistical machine translation, and Rao and Tetreault (2018) employ sequence-to-sequence (Seq2Seq) neural networks. Formality style transfer, however, is different from rich-resource machine translation mainly in two aspects. First, machine translation systems usually require massive parallel data for training, whereas the par- allel corpus for formality style transfer is much smaller. The WMT-17 English-Chinese corpus, for example, has 25M pairs (Bojar et al., 2017), but GYAFC has only ∼50K for each domain. Second, a formality-transferred sentence closely resembles the original one. As shown in Table 1, only a few words and punctuations are changed. It differs from machine translation, where the source and target sides are of completely different languages. As a result, directly applying a sequence-to-sequence (Seq2Seq) model not only has the risk of over- fitting to the training set, but also may fail to fully utilize the nature of informal and formal sentences. To tackle these problems, we propose a novel model S2S-SLS, a Sequence-to-Sequence …

Content truncated to 500 words