Fusing Label Embedding into BERT: An Efficient Improvement for Text Classification

Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021, pages 1743–1750 August 1–6, 2021. ©2021 Association for Computational Linguistics 1743 Fusing Label Embedding into BERT: An Efficient Improvement for Text Classification Yijin Xiong, Yukun Feng, Hao Wu, Hidetaka Kamigaito, and Manabu Okumura Institute of Innovative Research, Tokyo Institute of Technology {yijinx,yukun,wuhao,kamigaito,oku}@lr.pi.titech.ac.jp Abstract With pre-trained models, such as BERT, gain- ing more and more attention, plenty of re- search has been done to further promote their capabilities, from enhancing the experimental procedures (Sun et al., 2019) to improving the mathematical principles. In this paper, we pro- pose a concise method for improving BERT’s performance in text classification by utilizing a label embedding technique while keeping al- most the same computational cost. Experimen- tal results on six text classification benchmark datasets demonstrate its effectiveness. 1 Introduction Text classification is a classic problem in natural language processing (NLP). The task is to anno- tate a predefined class or classes to a given text, where text representation is an important interme- diate step. A variety of neural models have been developed to learn better text representations, including con- volution models (Kim, 2014; Kalchbrenner et al., 2014; Zhang et al., 2015; Conneau et al., 2017; Johnson and Zhang, 2017; Zhang et al., 2017; Shen et al., 2018), recurrent models (Liu et al., 2016; Yogatama et al., 2017; Seo et al., 2017; Wang et al., 2018b), and attention mechanisms (Yang et al., 2016; Lin et al., 2017). Pre-trained models have also been greatly bene- ficial in text classification in that they help stream- line the training process by avoiding a start from zero (Stein et al., 2019; Wang et al., 2017; Jiang et al., 2019). One group of approaches has focused on word embeddings, such as word2vec (Mikolov et al., 2013) and GloVe (Pennington et al., 2014); another has focused on contextualized word em- beddings, from CoVe (McCann et al., 2017) to ELMo (Peters et al., 2018), OpenAI GPT (Radford et al., 2018), ULMFiT (Howard and Ruder, 2018), and BERT (Devlin et al., 2019). BERT has achieved particularly impressive per- formances across a variety of NLP tasks. With its success, models pre-trained on a large amount of data, such as ERNIE (Zhang et al., 2019), RoBERTa (Liu et al., 2019), UniLM (Dong et al., 2019), and XLnet (Yang et al., 2019), have become popular thanks to their ability in learning contex- tualized representations. These models are based on the multi-layered bidirectional attention mecha- nism (Vaswani et al., 2017) and are trained through the masked word prediction task, which are two of the main components of BERT. Continuing to in- vestigate the potential of BERT remains important, since the findings can help with the investigation of variants of BERT as well. In this work, we propose a simple but effective method to improve BERT’s performance in text classification. We enhance the contextual represen- tation learning through encoding the texts of class labels (e.g. “world”, “sports”, “business”, and “sci- ence technology” in the AGNews dataset) along with the …

Content truncated to 500 words