Improving Formality Style Transfer with Context-Aware Rule Injection

Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing, pages 1561–1570 August 1–6, 2021. ©2021 Association for Computational Linguistics 1561 Improving Formality Style Transfer with Context-Aware Rule Injection Zonghai Yao University of Massachusetts Amherst zonghaiyao@cs.umass.edu Hong Yu University of Massachusetts Lowell hong yu@uml.edu Abstract Models pre-trained on large-scale regular text corpora often do not work well for user- generated data where the language styles dif- fer significantly from the mainstream text. Here we present Context-Aware Rule Injec- tion (CARI), an innovative method for formal- ity style transfer (FST). CARI injects multi- ple rules into an end-to-end BERT-based en- coder and decoder model. It learns to select optimal rules based on context. The intrin- sic evaluation showed that CARI achieved the new highest performance on the FST bench- mark dataset. Our extrinsic evaluation showed that CARI can greatly improve the regular pre- trained models’ performance on several tweet sentiment analysis tasks. 1 Introduction Many user-generated data deviate from standard language in vocabulary, grammar, and language style. For example, abbreviations, phonetic sub- stitutions, Hashtags, acronyms, internet language, ellipsis, and spelling errors, etc are common in tweets (Ghani et al., 2019; Muller et al., 2019; Han et al., 2013; Liu et al., 2020). Such irregularity leads to a significant challenge in applying existing language models pre-trained on large-scale corpora dominated with regular vocabulary and grammar. One solution is using formality style transfer (FST) (Rao and Tetreault, 2018), which aims to transfer the input text’s style from the informal domain to the formal domain. This may improve the down- stream NLP applications such as information ex- traction, text classification and question answering. A common challenge for FST is low resource (Wu et al., 2020; Malmi et al., 2020; Wang et al., 2020). Therefore, approaches that integrate exter- nal knowledge, such as rules, have been developed. However, existing work (Rao and Tetreault, 2018; Wang et al., 2019) deploy context-insensitive rule injection methods (CIRI). As shown in Figure 1, when we try to use CIRI-based FST as the prepro- cessing for user-generated data in the sentiment classification task, according to the rule detection system, ”extro” has two suggested changes ”extra” or ”extrovert” and ”intro” corresponds to either ”in- troduction” or ”introvert.” The existing CIRI-based FST models would arbitrarily choose rules follow- ing first come first served (FCFS). As such, the input ”always, always they think I an extro, but Im a big intro actually” could be translated wrongly as ”they always think I am an extra, but actually, I am a big introduction.” This leads to the wrong senti- ment classification since the FST result completely destroys the original input’s semantic meaning. In this work, we propose Context-Aware Rule Injection (CARI), an end-to-end BERT-based en- coder and decoder model that is able to learn to select optimal rules based on context. As shown in Figure 1, CARI chooses rules based on context. With CARI-based FST, pre-trained models can per- form better on the …

Content truncated to 500 words