Improving historical spelling normalization with bi-directional LSTMs and multi-task learning

Proceedings of COLING 2016, the 26th International Conference on Computational Linguistics: Technical Papers, pages 131–139, Osaka, Japan, December 11-17 2016. Improving historical spelling normalization with bi-directional LSTMs and multi-task learning Marcel Bollmann Department of Linguistics Ruhr-Universit¨at Bochum Germany bollmann@linguistics.rub.de Anders Søgaard Dpt. of Computer Science University of Copenhagen Denmark soegaard@hum.ku.dk Abstract Natural-language processing of historical documents is complicated by the abundance of variant spellings and lack of annotated data. A common approach is to normalize the spelling of histori- cal words to modern forms. We explore the suitability of a deep neural network architecture for this task, particularly a deep bi-LSTM network applied on a character level. Our model compares well to previously established normalization algorithms when evaluated on a diverse set of texts from Early New High German. We show that multi-task learning with additional normalization data can improve our model’s performance further. 1 Introduction Interest in computational processing of historical documents is on the rise, as evidenced by the growing field of digital humanities and the increasing number of digitally available resources of historical data. Spelling normalization, i.e. the mapping of historical spelling variants to standardized/modernized forms, is often employed as a pre-processing step to allow the utilization of existing tools for the respective modern target language (Piotrowski, 2012). Training data for supervised learning of spelling normalization is typically scarce in the historical domain. Furthermore, dialectal influences and even individual preferences of an author can have a huge impact on the spelling characteristics in a particular text, meaning that even training data from other corpora of the same language and time period cannot always be reliably used. Algorithms have often been developed with this fact in mind, e.g. by being based on some form of phonetic, graphematic, or semantic similarity measure (Jurish, 2010; Bollmann, 2012; Amoia and Martinez, 2013). On the other hand, neural networks – and particularly deep networks with several hidden layers – are assumed to work best when trained on large amounts of data. It is therefore not clear whether neural networks are a good choice for this particular domain. We frame spelling normalization as a character-based sequence labeling task, and explore the suit- ability of a deep bi-directional long short-term memory model (bi-LSTM) in this setting. By basing our model on individual characters as input, along with performing some basic preprocessing (e.g., down- casing all characters), we keep the vocabulary size small, which in turn reduces the model’s complexity and the amount of data required to train it effectively. We show that this model outperforms both the ex- isting normalization tool Norma (Bollmann, 2012) and a CRF-based tagger when evaluated on a diverse dataset from Early New High German. Furthermore, we experiment with a multi-task learning setup using auxiliary data that has similar, but not identical spelling characteristics to the target text. We show that using bi-LSTMs with this multi-task learning setup can improve normalization accuracy further, while Norma and CRF do not profit much from the additional data in a traditional setup. This …

Content truncated to 500 words