Bag-of-Words vs. Graph vs. Sequence in Text Classification: Questioning the Necessity of Text-Graphs and the Surprising Strength of a Wide MLP

Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics Volume 1: Long Papers, pages 4038 - 4051 May 22-27, 2022 c⃝2022 Association for Computational Linguistics Bag-of-Words vs. Graph vs. Sequence in Text Classification: Questioning the Necessity of Text-Graphs and the Surprising Strength of a Wide MLP Lukas Galke University of Kiel, Germany MPI for Psycholinguistics, the Netherlands lukas.galke@mpi.nl Ansgar Scherp University of Ulm, Germany ansgar.scherp@uni-ulm.de Abstract Graph neural networks have triggered a resur- gence of graph-based text classification meth- ods, defining today’s state of the art. We show that a wide multi-layer perceptron (MLP) us- ing a Bag-of-Words (BoW) outperforms the re- cent graph-based models TextGCN and Hete- GCN in an inductive text classification set- ting and is comparable with HyperGAT. More- over, we fine-tune a sequence-based BERT and a lightweight DistilBERT model, which both outperform all state-of-the-art models. These results question the importance of synthetic graphs used in modern text classifiers. In terms of efficiency, DistilBERT is still twice as large as our BoW-based wide MLP, while graph-based models like TextGCN require set- ting up an O(N 2) graph, where N is the vocabulary plus corpus size. Finally, since Transformers need to compute O(L2) atten- tion weights with sequence length L, the MLP models show higher training and inference speeds on datasets with long sequences. 1 Introduction Text categorization is the task of assigning topical categories to text units such as documents, social media postings, or news articles. Research on text categorization is a very active field as just the sheer amount of new methods in recent surveys shows (Bayer et al., 2021; Li et al., 2020; Zhou et al., 2020; Kowsari et al., 2019; Kadhim, 2019). There are approaches based on a Bag of Words (BoW) that perform text categorization purely on the basis of a multiset of tokens. Among them are Deep Averaging Networks (DAN) (Iyyer et al., 2015), a deep Multi-Layer Perceptron (MLP) model with n layers that relies on averag- ing the BoW, Simple Word Embedding Models (SWEM) (Shen et al., 2018) that explores different pooling strategies for pretrained word embeddings, and fastText (Bojanowski et al., 2017), which uses a linear layer on top of pretrained word embed- dings. These models count the occurrence of all tokens in the input sequence, while disregarding word position and order, and then rely on word em- beddings and fully connected feedforward layer(s). We call these BoW-based models. Among the most popular recent methods for text categorization are graph-based models such as TextGCN (Yao et al., 2019) that first induce a synthetic word-document co-occurence graph over the corpus and subsequently apply a graph neural network (GNN) to perform the classification task. Besides TextGCN, there are follow-up works like HeteGCN (Ragesh et al., 2021), TensorGCN (Liu et al., 2020), and HyperGAT (Ding et al., 2020), which we collectively call graph-based models. Finally, there is the well-known Trans- former (Vaswani et al., 2017) universe with models such as BERT (Devlin et al., 2019) and its size- reduced variants …

Content truncated to 500 words