Can Model Fusing Help Transformers in Long Document Classification? An Empirical Study

Proceedings of Recent Advances in Natural Language Processing, pages 871–878 Varna, Sep 4–6, 2023 https://doi.org/10.26615/978-954-452-092-2_094 871 Can Model Fusing Help Transformers in Long Document Classification? An Empirical Study Damith Premasiri♢, Tharindu Ranasinghe♡ and Ruslan Mitkov♠ ♢University of Wolverhampton, Wolverhampton, UK ♡Aston University, Birmingham, UK ♠Lancaster University, Lancaster, UK damith.premasiri@wlv.ac.uk, t.ranasinghe@aston.ac.uk r.mitkov@lancaster.ac.uk Abstract Text classification is an area of research which has been studied over the years in Natural Lan- guage Processing (NLP). Adapting NLP to mul- tiple domains has introduced many new chal- lenges for text classification and one of them is long document classification. While state- of-the-art transformer models provide excellent results in text classification, most of them have limitations in the maximum sequence length of the input sequence. The majority of the trans- former models are limited to 512 tokens, and therefore, struggle with long document classifi- cation problems. In this research, we explore the employment of Model Fusing for long docu- ment classification while comparing the results with well-known BERT and Longformer archi- tectures. 1 Introduction Text classification is one of the critical tasks in Nat- ural Language Processing, which refers to finding the suitable label/ labels to a particular input text (Kowsari et al., 2019; Miro´nczuk and Protasiewicz, 2018). It has a wide range of applications in dif- ferent domains such as sentiment analysis (Dang et al., 2020b,a), fake news detection (Thota et al., 2018; Kumar et al., 2020; Ahmad et al., 2020) and offensive language identification (Ranasinghe and Zampieri, 2020; Husain and Uzuner, 2021). These tasks are generally referred to as sentence classi- fication tasks since the input text is typically in the form of sentences. In recent years, transformer models such as BERT have provided state-of-the- art results in these text classification tasks (Ranas- inghe et al., 2019; Gaikwad et al., 2021). While most of the text classification tasks are sentence classification, several domains require classifying lengthy texts into labels typically re- ferred to as document classification. Specifically, domains such as legal and medical often contain long documents that need document classification methods (Chalkidis et al., 2019a; Hettiarachchi et al., 2023). However, adapting the transformer models that produced state-of-the-art results in sen- tence classification to document classification is challenging (Pappagari et al., 2019). The most common transformer models, such as BERT (De- vlin et al., 2019), have a limitation of 512 tokens in their input layer, which means the tokens in a lengthy document exceeding this limit will be trun- cated in the tokenisation step. The limitations outlined above have attracted significant attention from the research community, leading to the exploration of new document classifi- cation architectures. One widely adopted approach is to leverage transformer models that can process longer sequences. Notably, the Longformer (Belt- agy et al., 2020) and BigBird (Zaheer et al., 2020) transformer models have demonstrated exceptional performance in document classification tasks, with the capacity to accommodate up to 4,096 tokens. However, training transformer models that can pro- cess longer sequences is a resource-intensive task, and it may not be feasible …

Content truncated to 500 words