Evaluating the Noisy Channel Model for the Normalization of Historical Texts: Basque, Spanish and Slovene

Evaluating the Noisy Channel Model for the Normalization of Historical Texts: Basque, Spanish and Slovene Izaskun Etxeberria, I˜naki Alegria, Larraitz Uria, Mans Hulden University of the Basque Country, University of Colorado {izaskun.etxeberria i.alegria larraitz.uria}@ehu.eus, mans.hulden@colorado.edu Abstract This paper presents a method for the normalization of historical texts using a combination of weighted finite-state transducers and language models. We have extended our previous work on the normalization of dialectal texts and tested the method against a 17th century literary work in Basque. This preprocessed corpus is made available in the LREC repository. The performance of this (semi-)supervised method for learning relations between historical and contemporary word forms is evaluated against resources in three languages. The method we present learns to map phonological changes using a noisy channel model; it is a solution that uses a limited amount of supervision in order to achieve adequate performance without the need of an unrealistic amount of manual effort. The model is based on techniques commonly used for phonological inference and producing Grapheme-to-Grapheme conversion systems encoded as weighted transducers and produces F-scores above 80% in the task for Basque. A wider evaluation shows that the approach performs equally well with all the languages in our evaluation suite: Basque, Spanish and Slovene. A comparison against other methods that address the same task is also provided. Keywords: computational morphology, phonology, lexical normalization, historical texts, digital libraries 1. Introduction and scenario Historical documents are usually written in ancient lan- guages that exhibit a number of differences in comparison to modern languages, all of which have a significant impact on Natural Language Processing (NLP) (Piotrowski, 2012). Both historical and dialectal texts present similar problems from an NLP point of view in that NLP tools developed for contemporary standard language often fail in handling the linguistic varieties encountered in such texts. A majority of NLP tools are designed to process newspaper texts written in contemporary language. The processing of standardized modern languages exhibits some characteris- tics that are not found in historical and dialectal corpora. For example, (i) a standard variant is used for writing com- munication which is well documented by dictionaries and grammars; (ii) these languages have standard orthogra- phies and the majority of published texts adhere to these orthographic norms; (iii) large amounts of text are electron- ically available and can be used for developing NLP tools and resources. On the contrary, most of these characteris- tics are not shared by historical and dialectal text resources and, therefore, standard NLP tools can often not be directly applied to such corpora. Traditionally, some degree of lexical normalization is per- formed when working with historical and dialectal texts in order to link each variant to its corresponding standard form. Once the texts are normalized, standard NLP and IR (Information Retrieval) tools can be applied to the corpora with reasonably high performance. Canonicalization is the term used in this area, a term which referring to mapping each non-standard variant to a canonical one (Jurish, 2010). Accurate normalization can be very …

Content truncated to 500 words