Applying the Transformer to Character-level Transduction

Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics, pages 1901–1907 April 19 - 23, 2021. ©2021 Association for Computational Linguistics 1901 Applying the Transformer to Character-level Transduction Shijie WuZ Ryan CotterellQ,6 Mans HuldenX ZJohns Hopkins University 6University of Cambridge QETH Z¨urich XUniversity of Colorado Boulder shijie.wu@jhu.edu ryan.cotterell@inf.ethz.ch mans.hulden@colorado.edu Abstract The transformer (Vaswani et al., 2017) has been shown to outperform recurrent neural network-based sequence-to-sequence models in various word-level NLP tasks. character-level transduction tasks, e.g.Yet for mor- phological inflection generation and histori- cal text normalization, there are few works that outperform recurrent models using the transformer. In an empirical study, we un- cover that, in contrast to recurrent sequence- to-sequence models, the batch size plays a crucial role in the performance of the trans- former on character-level tasks, and we show that with a large enough batch size, the trans- former does indeed outperform recurrent mod- els. We also introduce a simple technique to handle feature-guided character-level trans- duction that further improves performance. With these insights, we achieve state-of-the-art performance on morphological inflection and historical text normalization. We also show that the transformer outperforms a strong base- line on two other character-level transduction tasks: grapheme-to-phoneme conversion and transliteration. 1 Introduction The transformer (Vaswani et al., 2017) has become a popular architecture for sequence-to-sequence transduction in NLP. It has achieved state-of-the- art performance on a range of common word-level transduction tasks: neural machine translation (Bar- rault et al., 2019), question answering (Devlin et al., 2019) and abstractive summarization (Dong et al., 2019). In addition, the transformer forms the back- bone of the widely-used BERT (Devlin et al., 2019). Yet for character-level transduction tasks like mor- phological inflection, the dominant model has re- mained a recurrent neural network-based sequence- Code will be available at https://github.com/ shijie-wu/neural-transducer. 16 32 64 128 256 512 Batch Size 76 78 80 82 84 86 88 90 ACC Wu and Cotterell (2019) Wu and Cotterell (2019) (Our Eval) Wu and Cotterell (2019) + LR Warmup Vanilla Transformer Feature Invariant Transformer Figure 1: Development set accuracy for 5 languages on morphological inflection with different batch sizes. We evince our two primary contributions: (1) we set the new state of the art morphological inflection using the transformer and (2) we demonstrate the transformer’s dependence on the batch size. to-sequence model with attention (Cotterell et al., 2018). This is not for lack of effort—but rather, it is the case that the transformer has consistently under- performed in experiments on average (Tang et al., 2018b).1 As anecdotal evidence of this, we note that in the 2019 SIGMORPHON shared task on cross-lingual transfer for morphological inflection, no participating system was based on the trans- former (McCarthy et al., 2019). Character-level transduction models are often trained with less data than their word-level coun- terparts: In contrast to machine translation, where millions of training samples are available, the 2018 SIGMORPHON shared task (Cotterell et al., 2018) high-resource setting only provides ≈ 10k training examples per language. It is …

Content truncated to 500 words