On the Cost-Effectiveness of Stacking of Neural and Non-Neural Methods for Text Classification: Scenarios and Performance Prediction

Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021, pages 4003–4014 August 1–6, 2021. ©2021 Association for Computational Linguistics 4003 On the Cost-Effectiveness of Stacking of Neural and Non-Neural Methods for Text Classification: Scenarios and Performance Prediction Christian Gomes UFMG - Brazil christianreis@dcc.ufmg.br Marcos Andr´e Gonc¸alves UFMG - Brazil mgoncalv@dcc.ufmg.br Leonardo Rocha UFSJ - Brazil lcrocha@ufsj.edu.br Sergio Canuto IFG - Brazil sergio.canuto@ifg.edu.br Abstract Neural network algorithms such as those based on transformers and attention models have ex- celled on Automatic Text Classification (ATC) tasks. However, such enhanced performance comes at high computational costs. Ensem- bles of simpler classifiers (i.e., stacking) that exploit algorithmic and representational complementarities have also been shown to produce top-notch performance in ATC, enjoying high effectiveness and potentially lower computational costs. In this context, we present the first and largest comparative study to exploit the cost-effectiveness of stacking of ATC classifiers, consisting of transformers and non-neural algorithms. In particular, we are interested in answering research questions such as: Is it possible to obtain an effective en- semble with significantly less computational cost than the best learning model for a given dataset? Disregarding the computational cost, can an ensemble improve the effectiveness of the best learning model? Besides answering such questions, another main contribution of this paper is the proposal of a low-cost oracle-based method that can predict the best ensemble in each scenario (with and without computational cost limitations) using only a fraction of the available training data. 1 Introduction Natural Language Processing, Machine Learning and Data Mining techniques work together to automate the fundamental task of Automatic Text Classification (ATC). ATC automatically associates documents with classes, providing means to organize information, allowing better comprehension and interpretation of the data. Algorithms based on neural networks (e.g., BERT (Devlin et al., 2018), XLNet (Yang et al., 2019)) have become the highlight in the area, where they are used both to learn features for text representation and as classification algorithms. The main problem of such methods is the very high computational costs needed for learning the model parameters (Sun et al., 2019; Cunha et al., 2021). Ensemble approaches, such as stacking, which combine the outputs of several base classification models to form an integrated output, have also been shown to excel in ATC (Dˇzeroski and ˇZenko, 2004; Ding and Wu, 2020), enjoying high effectiveness and computational costs that depend on the selected learning methods of the ensemble. They are motivated by the fact that distinct learning models or text representations may complement each other, uncovering specific structures that underlie the input/output relationship of the data. Early works (Larkey and Croft, 1996) aimed at showing combinations of different classification al- gorithms capable of producing better effectiveness results than any single type of classifier. However, the benefits of ensemble techniques against a strong classifier are not always clear (Yan- Shi Dong and Ke-Song Han, 2004), in part, due to the excellent generalization power of the best classifiers. In fact, previous ensemble works mostly focus on improving the overall classifica- tion effectiveness …

Content truncated to 500 words