Three Approaches to Client Email Topic Classification

Proceedings of Recent Advances in Natural Language Processing, pages 1015–1022 Varna, Sep 4–6, 2023 https://doi.org/10.26615/978-954-452-092-2_109 1015 Three Approaches to Client Email Topic Classification Branislava ˇSandrih Todorovi´c, Katarina Josipovi´c NLB DigIT / Serbia {branislava.sandrih.todorovic, katarina.kovacevic}@nlbdigit.rs Jurij Kodre NLB d.d. / Slovenia jurij.kodre@nlb.si Abstract This paper describes a use case that was imple- mented and is currently running in production at the Nova Ljubljanska Banka, that involves classifying incoming client emails in the Slove- nian language according to their topics and pri- orities. Since the proposed approach relies only on the Named Entity Recogniser (NER) of per- sonal names as a language-dependent resource (for the purpose of anonymisation), that is the only prerequisite for applying the approach to any other language. 1 Introduction Together with Nova Ljubljanska Banka’s (NLB) Centre of Excellence, Belgrade IT company NLB DigIT has a mission to incorporate smart, data-driven IT solutions to various aspects of every- day work in different Bank’s business sectors. One such case is the classification of client emails sent to the Bank’s Contact Centre (Kontakt Centar in Slovenian, dubbed KC) with respect to their topic (e.g. accounts/loans/cards, etc.) and priority (high, low, medium). In this paper we present the whole procedure, from having only the plain Outlook files to the models assigning topics and priorities to emails in real time. Section 2 mentions some of the previ- ously published scientific articles that inspired this research. We propose and discuss three different approaches for the modelling of emails in Section 3. Afterwards, we explain the process of preparing the dataset in Section 4: selection of optimal classifi- cation schemes and manual annotation, followed by certain cleaning steps and anonymisation of per- sonal information. Having the prepared dataset, we trained and exhaustively evaluated different NLP models for the case of topic classification. We explain the training process in more detail in Sec- tion 5, where we also show evaluation results first on a validation and then on a separate test set. Fi- nally, we close with some final remarks, ideas for further improvement and conclusions in Section 6. 2 Related Work The problem of email classification has been an active area of research for several decades, with nu- merous studies focusing on developing effective al- gorithms to accurately categorise incoming emails based on their content. Researchers still experi- ment with different techniques and approaches in order to improve the existing SPAM and Phishing email classifiers. Iqbal and Khan (2022) achieved the 98.06% accuracy using the binary Sup- port Vector Machine (SVM) classifier for the case of SPAM, whilst Shuaib et al. (2018) devel- oped the optimal SPAM classifier using Rota- tion Forest algorithm, achieving the accuracy of 94.2%. Ali et al. (2021) experimented with fea- ture engineering and RNN/CNN architectures, con- cluding that RNN provides the highest 94.9% accuracy. The binary SVM classifier also proved to be the best Phishing email detector for Sundararaj and Kul (2021), with 87.85% accu- racy. SVM proved to be the optimal classifier in all our experiments, which we clarify …

Content truncated to 500 words