Still not there? Comparing Traditional Sequence-to-Sequence Models to Encoder-Decoder Neural Networks on Monotone String Translation Tasks

Proceedings of COLING 2016, the 26th International Conference on Computational Linguistics: Technical Papers, pages 1703–1714, Osaka, Japan, December 11-17 2016. Still not there? Comparing Traditional Sequence-to-Sequence Models to Encoder-Decoder Neural Networks on Monotone String Translation Tasks Carsten Schnober†‡, Steffen Eger†, Erik-Lˆan Do Dinh†, and Iryna Gurevych†‡ †Ubiquitous Knowledge Processing Lab (UKP-TUDA) Department of Computer Science, Technische Universit¨at Darmstadt ‡Ubiquitous Knowledge Processing Lab (UKP-DIPF) German Institute for Educational Research and Educational Information https://www.ukp.tu-darmstadt.de/ Abstract We analyze the performance of encoder-decoder neural models and compare them with well- known established methods. The latter represent different classes of traditional approaches that are applied to the monotone sequence-to-sequence tasks OCR post-correction, spelling correc- tion, grapheme-to-phoneme conversion, and lemmatization. Such tasks are of practical relevance for various higher-level research fields including digital humanities, automatic text correction, and speech recognition. We investigate how well generic deep-learning approaches adapt to these tasks, and how they perform in comparison with established and more specialized meth- ods, including our own adaptation of pruned CRFs. 1 Introduction Encoder-decoder neural models (Sutskever et al., 2014) are a generic deep-learning approach to sequence-to-sequence translation (Seq2Seq) tasks. They encode an input sequence into a vector rep- resentation from which the decoder generates an output. These models have shown to achieve state-of- the-art or at least highly competitive results for various NLP tasks including machine translation (Cho et al., 2014), conversation modeling (Vinyals and Le, 2015), question answering (Yin et al., 2016), and, more generally, language correction (Schmaltz et al., 2016; Xie et al., 2016). We have noticed that, given the enormous interest currently surrounding neural architectures, recent re- search appears to somewhat over-enthusiastically praise the performance of encoder-decoder approaches for Seq2Seq tasks. For example, while the encoder-decoder G2P model by Rao et al. (2015) achieves an extremely low error rate on the CMUdict dataset (Kominek and Black, 2004), the neural architec- ture itself has a mediocre performance and only outperforms traditional models in combination with a weighted finite state transducer. Similarly, Faruqui et al. (2016) report on “par or better” performance of their inflection generation neural architecture. However, a closer inspection of their results suggests that their system is sometimes worse and sometimes better than traditional approaches. Here, we aim for a more balanced comparison on three exemplary monotone1 Seq2Seq tasks, namely spelling correction, G2P conversion, and lemmatization. Monotone Seq2Seq tasks such as morpho- logical analysis/lemmatization, grapheme-to-phoneme conversion (G2P) (Yao and Zweig, 2015; Rao et al., 2015), transliteration (Sherif and Kondrak, 2007), and spelling correction (Brill and Moore, 2000) have been fundamental problem classes in natural language processing (NLP) ever since the origins of the field. Their simplicity vis-`a-vis non-monotonic problems such as machine translation renders them as particularly tractable testbeds of technological progress. Unlike previous work, which has typically focussed on only one specific subproblem of monotone Seq2Seq tasks at a time, we consider model performances on three such tasks simultaneously. This leads to a more balanced view on the relative performance of different models. We compare three variants of encoder-decoder models …

Content truncated to 500 words