E-VarM: Enhanced Variational Word Masks to Improve the Interpretability of Text Classification Models

Proceedings of the 29th International Conference on Computational Linguistics, pages 1036–1050 October 12–17, 2022. 1036 E-VarM: Enhanced Variational Word Masks to Improve the Interpretability of Text Classification Models Ling Ge1, Chunming Hu1,2,∗, Guanghui Ma1, Junshuang Wu4, Junfan Chen1, Jihong Liu5, Hong Zhang3, Wenyi Qin1, Richong Zhang1 1 School of Computer Science and Engineering, Beihang University, Beijing, China 2 College of Software, , Beihang University , Beijing, China 3 National Computer Network Emergency Response Technical Team, Beijing, China 4 Beijing Jinghang Research Institute of Computing and Communication, Beijing, China 5 School of Mechanical Engineering and Automation, Beihang University, Beijing, China {geling, hucm}@buaa.edu.cn, {magh, chenjf, zhangrc}@act.buaa.edu.cn, zhangh@isc.org.cn, ryukeiko@buaa.edu.cn,wenyi.qin@hotmail.com Abstract Enhancing the interpretability of text classifi- cation models can help increase the reliabil- ity of these models in real-world applications. Currently, most researchers focus on extracting task-specific words from inputs to improve the interpretability of the model. The competitive approaches exploit the Variational Information Bottleneck (VIB) to improve the performance of word masking at the word embedding layer to obtain task-specific words. However, these approaches ignore the multi-level semantics of the text, which can impair the interpretability of the model, and do not consider the risk of rep- resentation overlap caused by the VIB, which can impair the classification performance. In this paper, we propose an enhanced variational word masks approach, named E-VarM, to solve these two issues effectively. The E-VarM com- bines multi-level semantics from all hidden lay- ers of the model to mask out task-irrelevant words and uses contrastive learning to readjust the distances between representations. Empiri- cal studies on ten benchmark text classification datasets demonstrate that our approach outper- forms the SOTA methods in simultaneously improving the interpretability and accuracy of the model. 1 Introduction With the widespread adoption of neural networks in text classification tasks (Bastings and Filippova, 2020; Halder et al., 2020; Schick et al., 2020; Lv et al., 2021), the classification models are expected to provide not only highly accurate classification results but also reasonable prediction rationales (Peake and Wang, 2018; Sun et al., 2021). These *Corresponding Author prediction rationales, which are short yet informa- tive parts of the input for classification predictions (Bastings et al., 2019), manifest the interpretability of the model. The better the interpretability of the model, the more reasonable rationales the model provides (Lin et al., 2021). Therefore, improving the interpretability of the model helps to boost the reliability of these classifiers in real-world applica- tions (Jacovi and Goldberg, 2020). To improve the interpretability of the classifiers, many methods have been proposed (Chrysostomou and Aletras, 2021; Bastings et al., 2019). Some studies rely on additional inputs at the training time, such as pre-defined information and human anno- tations (Erion et al., 2019; Plumb et al., 2020), all of which involve high human costs. Other works resort to assigning a binary Bernoulli variable to each input word with promising results, among which the competitive approaches are the Sparse- VIB (Paranjape et al., 2020) and Vmask (Chen and Ji, 2020) models . They all employ …

Content truncated to 500 words