ConvTextTM: An Explainable Convolutional Tsetlin Machine Framework for Text Classification

Proceedings of the 13th Conference on Language Resources and Evaluation (LREC 2022), pages 3761–3770 Marseille, 20-25 June 2022 © European Language Resources Association (ELRA), licensed under CC-BY-NC-4.0 3761 ConvTextTM: An Explainable Convolutional Tsetlin Machine Framework for Text Classification Bimal Bhattarai, Ole-Christoffer Granmo, Lei Jiao University of Agder Norway {bimal.bhattarai, ole.granmo, lei.jiao}@uia.no Abstract Recent advancements in natural language processing (NLP) have reshaped the industry, with powerful language models such as GPT-3 achieving superhuman performance on various tasks. However, the increasing complexity of such models turns them into “black boxes”, creating uncertainty about their internal operation and decision-making. Tsetlin Machine (TM) employs human-interpretable conjunctive clauses in propositional logic to solve complex pattern recognition problems and has demonstrated competitive performance in various NLP tasks. In this paper, we propose ConvTextTM, a novel convolutional TM architecture for text classification. While legacy TM solutions treat the whole text as a corpus-specific set-of-words (SOW), ConvTextTM breaks down the text into a sequence of text fragments. The convolution over the text fragments opens up for local position-aware analysis. Further, ConvTextTM eliminates the dependency on a corpus-specific vocabulary. Instead, it employs a generic SOW formed by the tokenization scheme of the Bidirectional Encoder Representations from Transformers (BERT) (Devlin et al., 2019a). The convolution binds together the tokens, allowing ConvTextTM to address the out-of-vocabulary problem as well as spelling errors. We investigate the local explainability of our proposed method using clause-based features. Extensive experiments are conducted on seven datasets, to demonstrate that the accuracy of ConvTextTM is either superior or comparable to state-of-the-art baselines. Keywords: Set-Of-Word (SOW), Convolutional Tsetlin Machine (CTM), Explainable, Human-Interpretable, Language Model, Text Classification, Tsetlin Machine (TM). 1. Introduction Recent advances in Artificial Intelligence (AI) have brought increasingly accurate learning algorithms and powerful computation platforms. However, the ac- curacy gains come with escalating computation costs, and models are getting too complicated for humans to comprehend. Prohibitive computation costs hinder the training of state-of-the-art models, while a lack of model transparency reduces trust. When end-users can- not understand model decisions and researchers report on how susceptible the models are to data bias, confi- dence in machine learning models declines. Mounting computation costs are problematic because of the long-term environmental impact. The obscurity of AI-driven decisions, however, raises immediate eth- ical concerns. Indeed, many decision errors can be fa- tal in high-risk application domains, including bank- ing (Le and Viviani, 2018), medicine (Esteva et al., 2019), bioinformatics (Freitas et al., 2008), and self- driving cars (Badue et al., 2020). Transparency is re- quired to ensure fair and safe decisions. AI model inter- pretability is thus a research area of increasing interest, crucial for further advancement of the AI field. The field of Natural Language Processing (NLP) man- ifests all of the above concerns. With the develop- ment of powerful language models such as Transform- ers (Vaswani et al., 2017), the prediction accuracy al- ready peaks for several NLP tasks, such as sentiment analysis and text classification. However, model com- plexity is immense, and only approximate interpreta- tion techniques …

Content truncated to 500 words