An Evaluation of Neural Machine Translation Models on Historical Spelling Normalization

Proceedings of the 27th International Conference on Computational Linguistics, pages 1320–1331 Santa Fe, New Mexico, USA, August 20-26, 2018. 1320 An Evaluation of Neural Machine Translation Models on Historical Spelling Normalization Gongbo Tang, Fabienne Cap, Eva Pettersson, and Joakim Nivre Uppsala University Uppsala, Sweden firstname.lastname@lingfil.uu.se Abstract In this paper, we apply different NMT models to the problem of historical spelling normaliza- tion for five languages: English, German, Hungarian, Icelandic, and Swedish. The NMT models are at different levels, have different attention mechanisms, and different neural network architec- tures. Our results show that NMT models are much better than SMT models in terms of character error rate. The vanilla RNNs are competitive to GRUs/LSTMs in historical spelling normaliza- tion. Transformer models perform better only when provided with more training data. We also find that subword-level models with a small subword vocabulary are better than character-level models. In addition, we propose a hybrid method which further improves the performance of historical spelling normalization. 1 Introduction With increasing access to digital historical text, the processing of these historical texts is attracting more and more interest. However, in contrast to modern text, historical text processing faces more challenges. First, for historical text, there is little annotated data for training a model, which leads to data sparsity issues when using statistical methods, similar to the situation for low-resource languages. Second, there are a lot of variations in historical texts from different time periods, not only in spelling but also in lexical semantics and syntax. Therefore, the NLP tools developed for modern text cannot be used for these historical texts directly. Spelling normalization is the task of mapping a historical spelling to its modern spelling. It is usually used as a preprocessing step before feeding the historical text into modern NLP tools (Pettersson et al., 2013b; Bollmann, 2013; Sánchez-Martínez et al., 2013), which leads to much better results compared to analyzing unnormalized historical texts. There are some papers in which neural machine translation (NMT) models are employed for the spelling normalization task. Korchagina (2017) utilizes a character-level NMT model for medieval Ger- man texts. Bollmann et al. (2017) apply an attention-based NMT model to historical German texts. The evidence so far is too incomplete to draw any general conclusions about the utility of different NMT models for historical spelling normalization. We are interested in exploring how different properties of NMT models interact with different aspects of the spelling normalization problem and find some gener- alizations about the use of NMT models for this task. In this paper, we apply different NMT models to the spelling normalization task for historical stages of five languages, English, German, Hungarian, Icelandic, and Swedish. We compare our result to those of Pettersson et al. (2014), which are obtained with statistical machine translation (SMT) models. We investigate whether NMT models outperform SMT models in general, and explore which properties of NMT models are suitable for spelling normalization. Compared to the conventional machine translation (MT) tasks, we train models on token pairs instead …

Content truncated to 500 words