Bag & Tag’em - A New Dutch Stemmer

Proceedings of the 12th Conference on Language Resources and Evaluation (LREC 2020), pages 3868–3876 Marseille, 11–16 May 2020 c⃝ European Language Resources Association (ELRA), licensed under CC-BY-NC 3868 Bag & Tag’em - A New Dutch Stemmer Anne Jonker1, Corn´e de Ruijt1 and Jornt R. de Gruijl2 Vrije Universiteit Amsterdam, Faculty of Science 1, Bright & Company2 annejonker@gmail.com, c.a.m.de.ruijt@vu.nl, jornt.de.gruijl@brightcompany.nl Abstract We propose a novel stemming algorithm that is both robust and accurate compared to state-of-the-art solutions, yet addresses several of the problems that current stemmers face in the Dutch language. The main issue is that most current stemmers cannot handle 3rd person singular forms of verbs and many irregular words and conjugations, unless a (nearly) brute-force approach is used. Our algorithm combines a new tagging module with a stemmer that uses tag-specific sets of rigid rules: the Bag & Tag’em (BT) algorithm. The tagging module is developed and evaluated using three algorithms: Multinomial Logistic Regression (MLR), Neural Network (NN) and Extreme Gradient Boosting (XGB). The stemming module’s performance is compared with that of current state-of-the-art stemming algorithms for the Dutch Language. Even though there is still room for improvement, the new BT algorithm performs well in the sense that it is more accurate than the current stemmers and faster than brute-force-like algorithms. The code and data used for this paper can be found at: https://github.com/Anne-Jonker/Bag-Tag-em Keywords: Stemming, PoS tagging, Dutch 1. Introduction The increased availability of both text data and computa- tional power has resulted in an increase in applications us- ing text mining for various purposes: it has been applied in sentiment analyses to determine the overall reputation of a company (Pang et al., 2008), identifying and analysing cybercrime activities (Kontostathis et al., 2010) and fraud detection (Phua et al., 2010), to name a few. In order to perform such analyses and develop applications based on unstructured text data, data is commonly prepro- cessed in a number of ways. One such common preprocess- ing step in text mining is to reduce the number of variants of a word to a common root in order to reduce noise and en- hance the accuracy of the analyses that follow. This process of reducing variants of words to a common stem is referred to as stemming. Stemming has been studied thoroughly for the English language and considerable success has already been achieved. This is commonly done through use of the Porter algorithm (Porter and others, 1980) that uses a rigid rule-based system to stem a word. The problem with many other languages, including the Dutch language, is that they have been less of a focal point for research and that many words are irregular when it comes to their inflections. As such, they do not conform to the otherwise established rule sets, forming exceptions. Our aim was to construct a new algorithm that is both more accurate and robust than the state-of-the-art algorithms cur- rently in use for the Dutch language. Our algorithm, the Bag & Tag’em (BT) algorithm, achieves this through …

Content truncated to 500 words