Multi-pretraining for Large-scale Text Classification

Findings of the Association for Computational Linguistics: EMNLP 2020, pages 2041–2050 November 16 - 20, 2020. c⃝2020 Association for Computational Linguistics 2041 Multi-pretraining for Large-scale Text Classification Kang-Min Kim1 Bumsu Hyeon1 Yeachan Kim2 Jun-Hyung Park1 SangKeun Lee1,3 1Department of Computer Science and Engineering, Korea University, Seoul, Republic of Korea 2Artificial Intelligence Research Institute, Republic of Korea 3Department of Artificial Intelligence, Korea University, Seoul, Republic of Korea {kangmin89, bshyeon, yeachan, irish07, yalphy}@korea.ac.kr Abstract Deep neural network-based pretraining meth- ods have achieved impressive results in many natural language processing tasks including text classification. However, their applicabil- ity to large-scale text classification with numer- ous categories (e.g., several thousands) is yet to be well-studied, where the training data is insufficient and skewed in terms of categories. In addition, existing pretraining methods usu- ally involve excessive computation and mem- ory overheads. In this paper, we develop a novel multi-pretraining framework for large- scale text classification. This multi-pretraining framework includes both a self-supervised pre- training and a weakly supervised pretraining. We newly introduce an out-of-context words detection task on the unlabeled data as the self-supervised pretraining. It captures the topic-consistency of words used in sentences, which is proven to be useful for text classi- fication. In addition, we propose a weakly supervised pretraining, where labels for text classification are obtained automatically from an existing approach. Experimental results clearly show that both pretraining approaches are effective for large-scale text classification task. The proposed scheme exhibits signifi- cant improvements as much as 3.8% in terms of macro-averaging F1-score over strong pre- training methods, while being computationally efficient. 1 Introduction Large-scale text classification is a natural language processing (NLP) task and a long-standing yet chal- lenging problem. It seeks to classify arbitrary texts into semantically relevant classes or categories. A wide range of applications exploit large-scale text classification, including web search personal- ization (Chirita et al., 2005), contextual advertis- ing (Lee et al., 2013), topical web search (Broder et al., 2007), and recommender systems (Amini et al., 2015). The success of these applications is highly dependent on the quality of text classifica- tion. Many applications of large-scale text clas- sification require a sufficiently large taxonomy of topical categories to capture various topics in arbi- trary texts. In addition, it is necessary to collect a large amount of training data for each category in the taxonomy. Deep neural models have demonstrated promis- ing results in text classification tasks (Kim, 2014; Zhang et al., 2015; Howard and Ruder, 2018), ow- ing to their strong expressive power and less re- quirement for feature engineering. However, the deeper and more complex the neural model, the more it is essential for them to be trained on sub- stantial amount of training data. Hence, pretraining methods have attracted significant attention (Rad- ford et al., 2018; Devlin et al., 2019); these meth- ods leverage large text corpora to train a model with better generalization properties (Ruder et al., 2019). Recently, several pretraining methods us- ing bidirectional long short-term memory (LSTM) networks (Hochreiter and Schmidhuber, 1997) …

Content truncated to 500 words