Harnessing Pre-Trained Neural Networks with Rules for Formality Style Transfer

Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing, pages 3573–3578, Hong Kong, China, November 3–7, 2019. c⃝2019 Association for Computational Linguistics 3573 Harnessing Pre-Trained Neural Networks with Rules for Formality Style Transfer Yunli Wang†, Yu Wu♦, Lili Mou‡, Zhoujun Li†∗, Wenhan Chao† †State Key Lab of Software Development Environment, Beihang University, Beijing, China ♦Microsoft Research, Beijing, China ‡Dept. of Computing Science, University of Alberta, Edmonton, Canada {wangyunli,lizj,chaowenhan}@buaa.edu.cn Wu.Yu@microsoft.com doublepower.mou@gmail.com Abstract Formality text style transfer plays an impor- tant role in various NLP applications, such as non-native speaker assistants and child educa- tion. Early studies normalize informal sen- tences with rules, before statistical and neu- ral models become a prevailing method in the field. While a rule-based system is still a common preprocessing step for formality style transfer in the neural era, it could introduce noise if we use the rules in a na¨ıve way such as data preprocessing. To mitigate this prob- lem, we study how to harness rules into a state- of-the-art neural network that is typically pre- trained on massive corpora. We propose three fine-tuning methods in this paper and achieve a new state-of-the-art on benchmark datasets. 1 Introduction Text formality research is essential for a wide range of NLP applications, such as non-native speaker assistants and child education. Due to the progress of deep learning techniques, re- searchers make a step from formality understand- ing to formality-aware text generation. Recently, Rao and Tetreault (2018) published a dataset, the Grammarly’s Yahoo Answers Formality Corpus (GYAFC), serving as a benchmark testbed for for- mality style transfer, which aims to generate a for- mal sentence given an informal one, while keeping its semantic meaning. Since the GYAFC dataset is small, existing studies have realized the importance of rules as a preprocessing step of informal text, typically han- dling capitalization (e.g., “ARE YOU KIDDING ME?”), character repetition (e.g., “noooo”), slang words (e.g., “wanna”), etc. While rule-based pre- processing could largely simplify the formality ∗ Corresponding author. Our code and output are available at: https://github.com/jimth001/ formality_emnlp19.git Original informal sentence: I LOVE HIP-HOP , RAP , ROCK & POP BUT MY FAV MUSIC IS R & B Output of a rule-based system: I love hip-hop , rap , rock and pop but my fav music is r and b Table 1: Example of informal sentence and the output of a rule-based system. style transfer task, we observe that it also intro- duces noise, with an example shown in Table 1. Given a sentence with all capital letters, a common rule-based method would lower all characters ex- cept the first one. Some entities, such as R & B, are changed to lower case incorrectly, especially if not recognized as a proper noun. Another intuition of ours is that, due to the small size of the parallel corpus, it would be beneficial to leverage a large neural network, which is pre- trained on a massive corpus and has learned gen- eral …

Content truncated to 500 words