TNT: Text Normalization based Pre-training of Transformers for Content Moderation

Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing, pages 4735–4741, November 16–20, 2020. c⃝2020 Association for Computational Linguistics 4735 TNT: Text Normalization based Pre-training of Transformers for Content Moderation Fei Tan, Yifan Hu, Changwei Hu, Keqian Li, Kevin Yen Yahoo Research, New York, NY, USA {fei.tan, yifanhu, changweih, keqian.li, kevinyen}@verizonmedia.com Abstract In this work, we present a new language pre-training model TNT (Text Normalization based pre-training of Transformers) for con- tent moderation. Inspired by the masking strat- egy and text normalization, TNT is developed to learn language representation by training transformers to reconstruct text from four op- eration types typically seen in text manipula- tion: substitution, transposition, deletion, and insertion. Furthermore, the normalization in- volves the prediction of both operation types and token labels, enabling TNT to learn from more challenging tasks than the standard task of masked word recovery. As a result, the ex- periments demonstrate that TNT outperforms strong baselines on the hate speech classifica- tion task. Additional text normalization experi- ments and case studies show that TNT is a new potential approach to misspelling correction. 1 Introduction Language model pre-training (self-supervised or unsupervised learning) has been a popular thread in Natural Language Understanding (NLP) stud- ies recently due to its universal representation ca- pacity (Radford et al., 2018; Devlin et al., 2019; Radford et al., 2019; Brown et al., 2020). It has been thus widely used in a multitude of language processing tasks such as named entity recognition, sentiment analysis, question answering and con- tent moderation (Bodapati et al., 2019). In addi- tion, the masking pre-training paradigm introduced by BERT (Bidirectional Encoder Representations from Transformers) (Devlin et al., 2019) has been employed for other tasks such as image processing (Trinh et al., 2019), optical flow (Liu et al., 2019a), and audio-visual co-segmentation (Rouditchenko et al., 2019). Recently, many variants have been proposed to further improve the pre-training procedure (Liu et al., 2019b; Wang et al., 2019; Sun et al., 2019). They have also advanced the state-of-the-art perfor- mance on multiple downstream natural language understanding tasks (Leaderboard) consistently. Al- most all these studies train language models by predicting the masked words in different manners. The underlying mechanism is Cloze task (Taylor, 1953). The pre-training model itself, however, has not been fully exploited to address complicated yet feasible tasks. It is reasonable to expect that models can learn a better universal language representation if the pre-training procedure can be aligned with more challenging tasks. In this article, we attempt to improve the lan- guage representation by proposing TNT: Text Normalization based pre-training of Transformers. TNT enhances the language learning by utilizing text normalization pre-training objective, inspired by misspelling correction. Specifically, TNT ran- domly manipulates tokens from the input text. The objective is then to reconstruct the original tokens of the manipulated words based on the context by predicting both recovery operation type and orig- inal token labels as illustrated in Fig. 1. Unlike the masked language model, TNT has to offer two predictions to reconstruct …

Content truncated to 500 words