Corpus-Based Diacritic Restoration for South Slavic Languages

Corpus-Based Diacritic Restoration for South Slavic Languages Nikola Ljubeˇsi´c,∗‡ Tomaˇz Erjavec,∗ Darja Fiˇser†∗ ∗ Department of Knowledge Technologies, Joˇzef Stefan Institute Jamova cesta 39, SI-1000 Ljubljana, Slovenia tomaz.erjavec@ijs.si † Faculty of Arts, University of Ljubljana Aˇskerˇceva cesta 2, SI-1000 Ljubljana, Slovenia darja.fiser@ff.uni-lj.si ‡ Dept. of Information and Communication Sciences, University of Zagreb Ivana Luˇci´ca 3, HR-10000 Zagreb, Croatia nikola.ljubesic@ijs.si Abstract In computer-mediated communication, Latin-based scripts users often omit diacritics when writing. Such text is typically easily under- standable to humans but very difficult for computational processing because many words become ambiguous or unknown. Letter-level approaches to diacritic restoration generalise better and do not require a lot of training data but word-level approaches tend to yield better results. However, they typically rely on a lexicon which is an expensive resource, not covering non-standard forms, and often not available for less-resourced languages. In this paper we present diacritic restoration models that are trained on easy-to-acquire cor- pora. We test three different types of corpora (Wikipedia, general web, Twitter) for three South Slavic languages (Croatian, Serbian and Slovene) and evaluate them on two types of text: standard (Wikipedia) and non-standard (Twitter). The proposed approach considerably outperforms charlifter, so far the only open source tool available for this task. We make the best performing systems freely available. Keywords: computer-mediated communication, diacritic restoration, South-Slavic languages 1. Background and Motivation In computer-mediated communication, such as emails, in- stant messages, tweets etc. users of Latin-based scripts often replace characters with diacritics with their ASCII equivalents for ergonomic reasons, especially when typing on tablets and smartphones. Such text is typically easily un- derstandable to humans but very difficult for computational processing because many words without the diacritics are ambiguous or unknown. This is why diacritics restoration is an active research area. There are two main approaches to diacritic restoration: letter-level and word-level (Mihalcea and Nastase, 2002). While the letter-level approaches generalise better and therefore do not require a lot of training data (Tufis¸ and Ceaus¸u, 2008), word-level approaches (ˇSanti´c et al., 2009; Nov´ak and Sikl´osi, 2015) tend to yield better results, espe- cially for languages where the diacritics have a grammati- cal and/or semantic role. Most approaches in this category rely on a lexicon (Yarowsky, 1999; Tufis¸ and Chit¸u, 1999; ˇSanti´c et al., 2009) but these are expensive resources that are often not available, especially for less-resourced lan- guages. Additionally, these approaches do not deal with word forms not covered by the lexicon, an issue that is all the more accentuated in non-standard text. Other ap- proaches require reliable morphological, syntactic or se- mantic tools for the language at hand, at the very least a PoS tagger (Yarowsky, 1994; Simard, 1998; Tufis¸ and Chit¸u, 1999), still an obstacle for many lesser resourced languages or language varieties, such as nonstandard user-generated content. Recently, statistical machine translation was also used for diacritic restoration of Hungarian (Nov´ak and Sikl´osi, 2015). However, we consider the method to be too com- plex for the task at hand as diacritic restoration does not require …

Content truncated to 500 words