Exploring the Relationship Between Algorithm Performance, Vocabulary, and Run-Time in Text Classification

Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 3069–3082 June 6–11, 2021. ©2021 Association for Computational Linguistics 3069 Exploring the Relationship Between Algorithm Performance, Vocabulary, and Run-Time in Text Classification Wilson Fearn Brigham Young University wilson.fearn@gmail.com Orion Weller Brigham Young University oweller@byu.edu Kevin Seppi Brigham Young University kseppi@byu.edu Abstract Text classification is a significant branch of natural language processing, and has many applications including document classification and sentiment analysis. Unsurprisingly, those who do text classification are concerned with the run-time of their algorithms, many of which depend on the size of the corpus’ vocab- ulary due to their bag-of-words representation. Although many studies have examined the ef- fect of preprocessing techniques on vocabu- lary size and accuracy, none have examined how these methods affect a model’s run-time. To fill this gap, we provide a comprehensive study that examines how preprocessing tech- niques affect the vocabulary size, model per- formance, and model run-time, evaluating ten techniques over four models and two datasets. We show that some individual methods can re- duce run-time with no loss of accuracy, while some combinations of methods can trade 2- 5% of the accuracy for up to a 65% reduction of run-time. Furthermore, some combinations of preprocessing techniques can even provide a 15% reduction in run-time while simultane- ously improving model accuracy.1 1 Introduction With the increasing amount of text data available, text analysis has become a significant part of ma- chine learning (ML). Many problems in text analy- sis use ML methods to perform their task, ranging from classical problems like text classification and topic modeling, to more complex tasks like ques- tion answering. Although neural networks have become increasingly common in the research field, many industry NLP problems can be well served by less complex but more efficient and explainable models, such as Support Vector Machines (SVMs) or K-Nearest Neighbors (K-NN). 1Our code and results are publicly available at https: //github.com/wfearn/preprocessing-paper We focus on the text classification problem, where the dominant approach to using these non- neural models is to first calculate the number of unique terms in the dataset (the vocabulary, size V ) and encode each instance of the dataset into a bag-of-words (BoW) representation (Joachims, 1998; Zhang et al., 2010). This results in a high-dimensional vector of size V that indicates whether each given word of the vocabulary was used in this instance. However, the vanilla approach to the BoW representation can lead to sub-par performance, as shown by numerous studies that have exam- ined how preprocessing techniques affect the BoW w.r.t. performance and vocabulary size. These studies have examined this representation in fields such as information retrieval (Chaudhari et al., 2015; Patil and Atique, 2013; Beil et al., 2002; Senuma, 2011), text classification (Yang and Ped- ersen, 1997; Caragea et al., 2012; Uysal and Gu- nal, 2014; Vijayarani et al., 2015; Kumar and Har- ish, 2018; HaCohen-Kerner et al., 2020; Symeoni- dis et al., 2018) and topic modeling …

Content truncated to 500 words