Learning attention for historical text normalization by learning to pronounce

Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics, pages 332–344 Vancouver, Canada, July 30 - August 4, 2017. c⃝2017 Association for Computational Linguistics https://doi.org/10.18653/v1/P17-1031 Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics, pages 332–344 Vancouver, Canada, July 30 - August 4, 2017. c⃝2017 Association for Computational Linguistics https://doi.org/10.18653/v1/P17-1031 Learning attention for historical text normalization by learning to pronounce Marcel Bollmann Department of Linguistics Ruhr-Universität Bochum Germany bollmann@linguistics.rub.de Joachim Bingel Dept. of Computer Science University of Copenhagen Denmark bingel@di.ku.dk Anders Søgaard Dept. of Computer Science University of Copenhagen Denmark soegaard@di.ku.dk Abstract Automated processing of historical texts often relies on pre-normalization to mod- ern word forms. Training encoder-decoder architectures to solve such problems typi- cally requires a lot of training data, which is not available for the named task. We ad- dress this problem by using several novel encoder-decoder architectures, including a multi-task learning (MTL) architecture us- ing a grapheme-to-phoneme dictionary as auxiliary data, pushing the state-of-the- art by an absolute 2% increase in perfor- mance. We analyze the induced models across 44 different texts from Early New High German. Interestingly, we observe that, as previously conjectured, multi-task learning can learn to focus attention during decoding, in ways remarkably similar to recently proposed attention mechanisms. This, we believe, is an important step to- ward understanding how MTL works. 1 Introduction There is a growing interest in automated process- ing of historical documents, as evidenced by the growing field of digital humanities and the in- creasing number of digitally available collections of historical documents. A common approach to deal with the high amount of variance often found in this type of data is to perform spelling nor- malization (Piotrowski, 2012), which is the map- ping of historical spelling variants to standard- ized/modernized forms (e.g. vnd → und ‘and’). Training data for supervised learning of histor- ical text normalization is typically scarce, mak- ing it a challenging task for neural architec- tures, which typically require large amounts of la- beled data. Nevertheless, we explore framing the spelling normalization task as a character-based sequence-to-sequence transduction problem, and use encoder–decoder recurrent neural networks (RNNs) to induce our transduction models. This is similar to models that have been proposed for neu- ral machine translation (e.g., Cho et al. (2014)), so essentially, our approach could also be considered a specific case of character-based neural machine translation. By basing our model on individual characters as input, we keep the vocabulary size small, which in turn reduces the model’s complexity and the amount of data required to train it effectively. Us- ing an encoder–decoder architecture removes the need for an explicit character alignment between historical and modern wordforms. Furthermore, we explore using an auxiliary task for which data is more readily available, namely grapheme-to- phoneme mapping (word pronunciation), to reg- ularize the induction of the normalization models. We propose several architectures, including multi-task learning architectures taking advantage of the auxiliary data, and evaluate them across 44 small datasets from Early New High …

Content truncated to 500 words