Complicate Then Simplify: A Novel Way to Explore Pre-trained Models for Text Classification

Proceedings of the 29th International Conference on Computational Linguistics, pages 1136–1145 October 12–17, 2022. 1136 Complicate then Simplify: A Novel Way to Explore Pre-trained Models for Text Classification Xu Zhang and Zejie Liu and Yanzheng Xiang and Deyu Zhou∗ School of Computer Science and Engineering, Key Laboratory of Computer Network and Information Integration, Ministry of Education, Southeast University, China {xuzhang123,liuzejie,yz−xiang,d.zhou}@seu.edu.cn Abstract In the developing context of pre-trained models (PTMs), the performance of text classification has been continuously improved by directly em- ploying the features generated by PTMs. How- ever, such a way might not fully explore the knowledge in PTMs as it is constrained by the difficulty of the task. Compared to a difficult task, the learning algorithms tend to saturate early on the simple task. Moreover, the native sentence representations derived from BERT are prone to be collapsed and directly employ- ing such representation for text classification might fail to fully capture discriminative fea- tures. In order to address these issues, in this paper we propose a novel framework for text classification which implements a two-stage training strategy. In the pre-training stage, aux- iliary labels are introduced to increase the task difficulties and to fully exploit the knowledge in the pre-trained model. In the fine-tuning stage, the textual representation learned in the pre-training stage is employed and the classi- fier is fine-tuned to obtain better classification performance. Experiments were conducted on six text classification corpora and the results showed that the proposed framework outper- formed several state-of-the-art baselines. 1 Introduction Text classification is a fundamental task in the field of natural language processing and is widely em- ployed in various tasks such as question answering, sentiment analysis, and information retrieval. With the continuous development of machine learning algorithms, especially the success of deep learning methods, text classification has been significantly improved, e.g. CNNs (Kim, 2014; Lai et al., 2019), RNNs (Chen et al., 2017; Zhang et al., 2020), BERTs (Cui et al., 2019, 2020; Sun et al., 2021), etc. Recently, pre-trained models have been shining in classification-based natural language processing tasks (Cui et al., 2019, 2020; Sun et al., 2021). ∗Corresponding author CH: …很不错的。如果下次去无锡，我还是会选择这里。 EN: …were very good. If I go to Wuxi next time, I would still choose this place. 很(very) 不错(good) 下次(next time) 会(will) 选择(choose) FEATURE WEIGHTS Ours BERT Figure 1: An Example from the ChnSentiCorp corpus. The advent of BERT has led to an effective en- hancement of textual feature representation. A se- ries of improved pre-trained models have been pro- posed, e.g. RoBERTa (Liu et al., 2019), MacBERT (Cui et al., 2020), ERNIE (Sun et al., 2020), Chine- seBERT (Sun et al., 2021). For example, ERNIE learns real-world semantic knowledge by model- ing words, entities and entity relationships in mas- sive amounts of data (Sun et al., 2020). Sun et al. (2021) proposed a large-scale Chinese pre-trained model that incorporates glyph and pinyin informa- tion. The above works improve the textual repre- sentation of the pre-trained model by introducing external knowledge, without fully exploring the …

Content truncated to 500 words