Automatic Diacritics Restoration for Hungarian

Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pages 2286–2291, Lisbon, Portugal, 17-21 September 2015. c⃝2015 Association for Computational Linguistics. Automatic Diacritics Restoration for Hungarian Attila Nov´ak1,2 and Borb´ala Sikl´osi2 1MTA-PPKE Hungarian Language Technology Research Group, 2P´azm´any P´eter Catholic University, Faculty of Information Technology and Bionics 50/a Pr´ater street, 1083 Budapest, Hungary {novak.attila, siklosi.borbala}@itk.ppke.hu Abstract In this paper, we describe a method based on statistical machine translation (SMT) that is able to restore accents in Hungarian texts with high accuracy. Due to the ag- glutination in Hungarian, there are always plenty of word forms unknown to a sys- tem trained on a fixed vocabulary. In or- der to be able to handle such words, we integrated a morphological analyzer into the system that can suggest accented word candidates for unknown words. We evalu- ated the system in different setups, achiev- ing an accuracy above 99% at the highest. 1 Introduction Due to clumsy mobile device interfaces and re- luctance of users to spend too much time entering their message, a great amount of text is generated in a format that lacks the diacritic marks normally used in the orthography of the language the text is written in. Whatever the causes for the missing accents are, NLP applications should be able to re- store or generate the accented version of such texts prior to any further syntactic or semantic process- ing to avoid upstream errors. In this paper, we aim at solving the problem of restoring accents in Hungarian texts with the com- bined application of a statistical machine transla- tion system and a morphological analyzer. Our method can be applied to any other languages that have an accurate morphological analyzer. 2 Related work For Hungarian, there have been some attempts at creating accent restoration systems. Zaink´o et al. (2000) and Mihalcea and Nastase (2002) are examples for ML approaches, where the correct places of diacritics are predicted from the immedi- ate grapheme-level context of the unaccented let- ter with an accuracy of 95%. Thus, unseen words can also be accented, but incorrect forms may also be introduced into the text. Dictionary-based ap- proaches rely on large text corpora and the distri- bution of the different accented forms. Zaink´o et al. (2000) report to have achieved a performance of 98% of accuracy with their dictionary-based method. Nevertheless, their system cannot rec- ognize unseen wordforms quite common in Hun- garian. N´emeth et al. (2000) have implemented a complex text processing system for TTS applica- tions, applying morphological and syntactic anal- ysis. The authors report that the performance of accent restoration depends very much on the per- formance of the analyzers (achieving 95% accu- racy at best). Neither the implementations nor the resources used in these systems have been made publicly available. A language-independent tool, Charlifter (Scan- nell, 2011), is based on statistical methods relying on a lexicon, a bigram contextual model and char- acter distributions built from a training corpus. Its performance on Hungarian with its pre-built mod- …

Content truncated to 500 words