Efficient Nearest Neighbor Emotion Classification with BERT-whitening

Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 4738 - 4745 December 7-11, 2022 ©2022 Association for Computational Linguistics Efficient Nearest Neighbor Emotion Classification with BERT-whitening Wenbiao Yin and Lin Shang∗ State Key Laboratory for Novel Software Technology Department of Computer Science and Technology Nanjing University, Nanjing, China wenbiaoyin@smail.nju.edu.cn, shanglin@nju.edu.cn Abstract Retrieval-based methods have been proven ef- fective in many NLP tasks. Previous methods use representations from the pre-trained model for similarity search directly. However, the sentence representations from the pre-trained model like BERT perform poorly in retriev- ing semantically similar sentences, resulting in poor performance of the retrieval-based meth- ods. In this paper, we propose KNN-EC, a simple and efficient non-parametric emotion classification (EC) method using nearest neigh- bor retrieval. We use BERT-whitening to get better sentence semantics, ensuring that nearest neighbor retrieval works. Meanwhile, BERT- whitening can also reduce memory storage of datastore and accelerate retrieval speed, solv- ing the efficiency problem of the previous meth- ods. KNN-EC average improves the pre-trained model by 1.17 F1-macro on two emotion clas- sification datasets. 1 Introduction Retrieval-based methods enhance model perfor- mance by extracting textual information related to the input from large training data. The model inter- polates the original model distribution and retrieved results’ distribution as the final distribution for out- put. Retrieval-based methods can often improve the performance as the model is exposed to related knowledge not present in the input(Wang et al., 2022). Retrieval-based methods have been success- fully applied to many tasks such as language mod- eling(Khandelwal et al., 2019; Guu et al., 2020), machine translation(Gu et al., 2018; Bapna and Fi- rat, 2019; Khandelwal et al., 2020; Zheng et al., 2021), question answering(Chen et al., 2017; Lee et al., 2019), and computer vision(Devlin et al., 2015; Gur et al., 2021). Retrieval-based methods are expressive, adapt- able, and interpretable. Here, we propose KNN- EC, a simple and efficient non-parametric emotion ∗Corresponding author classification (EC) method using nearest neighbor retrieval. KNN-EC can be added to any pre-trained emotion classification model without further train- ing. Pre-trained models such as BERT (Devlin et al., 2019), RoBERTa (Liu et al., 2019), XLNet (Yang et al., 2019) have been widely proven ef- fective for emotion classification, and they obtain state-of-the-art results on many emotion classifica- tion tasks. However, the sentence representations from the pre-trained model like BERT perform poorly in retrieving semantically similar sentences(Reimers and Gurevych, 2019). Previous works found that pre-trained models always induced a non-smooth anisotropic semantic space of sentences, which harmed its performance of semantic similarity. To address this issue, Gao et al. (2018) designed a novel way to mitigate the degeneration problem by regularizing the word embedding matrix. Li et al. (2020) proposed to transform the BERT sentence embedding distribution into a smooth and isotropic Gaussian distribution through normalizing flow, an invertible function parameterized by the neural net- work. Because the above methods require addi- tional parameters added to the neural network, this is not suitable to be added to KNN-EC to …

Content truncated to 500 words