The Impact of Text Normalization on Multiword Expressions Discovery in Persian

Proceedings of Recent Advances in Natural Language Processing, pages 929–939 Sep 1–3, 2021. https://doi.org/10.26615/978-954-452-072-4_106 929 The Impact of Text Normalization on Multiword Expressions Discovery in Persian Katarzyna Marszalek-Kowalewska YUKKA Lab AG / Berlin, Germany k.marszalek.kowalewska@gmail.com Abstract This paper evaluates normalization procedures of Persian text for a downstream NLP task - multiword expressions (MWEs) discovery. We discuss the challenges the Persian lan- guage poses for NLP and evaluate open-source tools that try to address these difficulties. The best-performing tool is later used in the main task - MWEs discovery. In order to discover MWEs, we use association measures and a subpart of the MirasText corpus. The results show that an F-score is 26% higher in the case of normalized input data. 1 Introduction The field of computational linguistics (CL) and its engineering domain of natural language processing (NLP) has exploded in recent years. It seems to continue to gain momentum because of a straight- forward reason: human civilization is drowning in data. In 2008, Google reported that the Web had one trillion pages.1 In 2016, the number was estimated to 130 trillions.2 International Data Cor- poration projects that by 2025, available data may expand to 175 zettabytes.3 Although these esti- mates include video, image data, and databases, most of it is plain old text. Unstructured data (also known as free-form text) comprises 70% - 80% of the data available on computer networks. The in- formation content of this resource is unavailable to authorities, businesses, and individuals unless hu- mans read these texts or devise some other means to derive information value from them. And this is where Natural Language Processing comes to the 1https://www.itpro.co.uk/604911/google-says-the-web- hits-a-trillion-pages 2https://medium.com/@MichelKiflen/google-has- indexed-130-trillion-pages-how-would-you-find-the-one- you-need-d0afa303d6f6 3https://www.networkworld.com/article/3325397/idc- expect-175-zettabytes-of-data-worldwide-by-2025.html game. NLP procedures can be applied to character- ize, interpret, or understand the information content of a free-form text, in other words, to unlock the potential of unstructured data. However, since the quality of the input data in- fluences the quality of the output, in most cases before the NLP pipeline uses it, this unstructured data needs to undergo certain cleaning and nor- malization tasks, e.g., removal of extra whitespace, substitution of acronyms, transformation of numer- ical information, accent removal, substitution of special characters and emoji, or normalization of date format. This paper addresses the problem of normalizing texts in the Persian language, which is 5th content language for the Web according to W3Tech.4 In particular, we will focus on the impact of Persian text normalization on one of the downstream NLP tasks - multiword expressions discovery. MWEs are very frequent in language and it has been proved that their proper treatment can make a significant impact on a number of other NLP tasks, e.g. lex- icography (Church and Hanks, 1990; Fellbaum, 2016), word sense disambiguation (Finlayson and Kulkarni, 2011), part-of-speech tagging and pars- ing (Baldwin et al., 2004), information retrieval (Newman et al., 2012), language learning (Chris- tiansen and Arnon, 2017), machine translation (Carpuat and Diab, 2010) or sentiment analysis (Berend, 2011; Williams et al., 2015). To the best of …

Content truncated to 500 words