A Large-Scale Comparison of Historical Text Normalization Systems

Proceedings of NAACL-HLT 2019, pages 3885–3898 Minneapolis, Minnesota, June 2 - June 7, 2019. c⃝2019 Association for Computational Linguistics 3885 A Large-Scale Comparison of Historical Text Normalization Systems Marcel Bollmann Department of Computer Science University of Copenhagen, Denmark marcel@di.ku.dk Abstract There is no consensus on the state-of-the- art approach to historical text normalization. Many techniques have been proposed, in- cluding rule-based methods, distance metrics, character-based statistical machine translation, and neural encoder–decoder models, but stud- ies have used different datasets, different eval- uation methods, and have come to different conclusions. This paper presents the largest study of historical text normalization done so far. We critically survey the existing literature and report experiments on eight languages, comparing systems spanning all categories of proposed normalization techniques, analysing the effect of training data quantity, and using different evaluation methods. The datasets and scripts are made publicly available. 1 Introduction1 Spelling variation is one of the key challenges for NLP on historical texts, affecting the performance of tools such as part-of-speech taggers or parsers and complicating users’ search queries on a cor- pus. Normalization is often proposed as a solu- tion; it is commonly defined as the mapping of his- torical variant spellings to a single, contemporary “normal form” as exemplified in Figure 1. Automatic normalization of historical texts has a long history, going back to at least Fix (1980). Earlier approaches often rely on hand-crafted al- gorithms tailored to one specific language, while more recent approaches have focused on super- vised machine learning, particularly character- based statistical machine translation (SMT) and its neural equivalent (NMT). However, no clear consensus has emerged about the state of the art for this task, with papers either reporting an ad- vantage for NMT (Hämäläinen et al., 2018), SMT 1This work largely builds upon the author’s doctoral thesis (Bollmann, 2018), the research for which was carried out at Ruhr-Universität Bochum, Germany. their thair thaire thear thayr theaire theiare theire þer þere þair theyr thir ther thar Figure 1: Historical text normalization exemplified: mapping variant spellings from historical English texts to their normalization ‘their’ (Domingo and Casacuberta, 2018), or language- specific algorithms (Schneider et al., 2017). More- over, the quantity of annotated training data varies considerably between studies, making it diffi- cult to obtain practical recommendations for new projects seeking to use normalization techniques. Contributions This paper aims to provide the most comprehensive evaluation and analysis of historical text normalization systems so far. Mo- tivated by a systematic review of previous work on this topic (Sec. 2), only publicly available nor- malization systems covering a wide range of pro- posed techniques are selected (Sec. 3) and eval- uated across a diverse collection of historical datasets covering eight languages (Sec. 4). This is followed by a detailed analysis of the effect of training data quantity and a critical discussion of evaluation methods for assessing normalization quality (Sec. 5). The datasets and code are made freely available whenever possible,2 along with detailed instruc- tions on how to reproduce the experiments. 2https://github.com/coastalcph/ histnorm; …

Content truncated to 500 words