A Novel Perspective to Look At Attention: Bi-level Attention-based Explainable Topic Modeling for News Classification

Findings of the Association for Computational Linguistics: ACL 2022, pages 2280 - 2290 May 22-27, 2022 c⃝2022 Association for Computational Linguistics A Novel Perspective to Look At Attention: Bi-level Attention-based Explainable Topic Modeling for News Classification Dairui Liu, Derek Greene and Ruihai Dong Insight Centre for Data Analytics, Dublin School of Computer Science, University College Dublin, Ireland dairui.liu@ucdconnect.ie {derek.greene, ruihai.dong}@ucd.ie Abstract Many recent deep learning-based solutions have adopted the attention mechanism in var- ious tasks in the field of NLP. However, the in- herent characteristics of deep learning models and the flexibility of the attention mechanism increase the models’ complexity, thus leading to challenges in model explainability. To ad- dress this challenge, we propose a novel prac- tical framework by utilizing a two-tier atten- tion architecture to decouple the complexity of explanation and the decision-making process. We apply it in the context of a news article clas- sification task. The experiments on two large- scaled news corpora demonstrate that the pro- posed model can achieve competitive perfor- mance with many state-of-the-art alternatives and illustrate its appropriateness from an ex- plainability perspective. We release the source code here1. 1 Introduction The attention mechanism is one of the most im- portant components in recent deep learning-based architectures in natural language processing (NLP). In the early stages of its development, the encoder- decoder models (Bahdanau et al., 2015; Xu et al., 2015) often adopted an attention mechanism to improve the performance achieved by capturing different areas of the input sequence when gener- ating an output in the decoding process to solve issues arising in encoding long-form inputs. Sub- sequently, researchers have applied the attention mechanism to large-scale corpora and developed a range of pre-trained language models (Kalyan et al., 2021), such as BERT (Devlin et al., 2019) and GPT-1 (Radford et al., 2018). This has yielded great progress across a range of NLP tasks, in- cluding sentiment analysis (Zhao et al., 2021) and news classification (Wu et al., 2021). However, the inherent characteristics of deep learning models 1https://github.com/Ruixinhua/BATM and the flexibility of the attention mechanism in- crease these models’ complexity, thus leading to challenges in model explainability. Today, there is still no consensus among re- searchers regarding whether attention-based mod- els are explainable in theory. Some researchers believe that attention weights may reflect the im- portance of features during the decision-making process and thus can provide an explanation of their operation if we visualize features according to their weight distribution (Luong et al., 2015; Lu et al., 2018). However, other researchers have disagreed with this hypothesis. For example, Jain and Wallance’s study demonstrated that learned attention weights are often uncorrelated with fea- ture importance (Jain and Wallace, 2019). Some researchers have supported this viewpoint (Serrano and Smith, 2019), but treated with skepticism by others (Wiegreffe and Pinter, 2019). In this paper, rather than validating the attention explainability theoretically, we propose a novel, practical explainable attention-based solution. In- spired by the idea of topic models (Blei et al., 2003), our proposed solution decouples the …

Content truncated to 500 words