Joint Persian Word Segmentation Correction and Zero-Width Non-Joiner Recognition Using BERT

Proceedings of the 28th International Conference on Computational Linguistics, pages 4612–4618 Barcelona, Spain (Online), December 8-13, 2020 4612 Joint Persian Word Segmentation Correction and Zero-Width Non-Joiner Recognition Using BERT Ehsan Doostmohammadi⋆, Minoo Nassajian⋆, Adel Rahimi† ⋆Sharif University of Technology, Tehran, Iran †Dathena Science Pte. Ltd., Singapore {e.doostm72,m.nassajian2016}@student.sharif.edu adel.rahimi@dathena.io Abstract Words are properly segmented in the Persian writing system; in practice, however, these writing rules are often neglected, resulting in single words being written disjointedly and multiple words written without any white spaces between them. This paper addresses the problems of word seg- mentation and zero-width non-joiner (ZWNJ) recognition in Persian, which we approach jointly as a sequence labeling problem. We achieved a macro-averaged F1-score of 92.40% on a care- fully collected corpus of 500 sentences with a high level of difficulty. 1 Introduction People who have worked with real-world data in Persian natural language processing (NLP) are familiar with the Persian writing system’s problematic properties, the most important of which are word seg- mentation and ZWNJ, both consequences of a phenomenon called “orthographic ligature”. In Persian, white space character (U+0020) is used to segment words in a sentence, and ZWNJ (U+200C) is used between free morphemes in compound words (or combining forms), and also between one or several free morphemes and one or several bound morphemes to stop joining letters from connecting into a ligature (Moghaddam, 2007). Although there is a set of rules regarding the use of white space and ZWNJ set by the regulatory body of the Persian language (Academy of Persian Language and Literature, 2015), only a few people follow them in writing formal Persian, let alone the informal language. Some common errors regarding the use of white space and ZWNJ are as follows: (1) not seg- menting words where the last letter of the first word is a non-joiner character (e.g., va ¯am¯ar[- e] marg|omir behtar az xeyli|h¯ast1 “and the mortality rate is better than many”); (2) using ZWNJ in- stead of white space which could be categorized as (a) excessive use of ZWNJ (e.g., for separating first and last names) and (b) ZWNJ key hit mistakenly as in many cell phone keyboards the space key and the ZWNJ key are adjacent to each other; (3) using space instead of ZWNJ (e.g. mi konam “I do”); (4) using neither of them (e.g. mikonam “I do”); and many other spontaneous errors. In comparison to European languages, such as English and German, word segmentation in Asian languages like Japanese, Chinese (Li et al., 2019), and Thai (Aroonmanakun, 2002; Tesprasit et al., 2003), is more complicated, because space is not specifically used as an orthographic word boundary delimiter (Khan et al., 2018b). For instance, in Vietnamese, space is not only used to separate words, but it is also applied to separate syllables (that can be considered as a meaningful word or as a part of multi-syllable words) that make up words (Huyen et al., 2008). The problem in Persian is not similar to that of the abovementioned languages, but …

Content truncated to 500 words