Towards Reasonably-Sized Character-Level Transformer NMT by Finetuning Subword Systems

Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing, pages 2572–2579, November 16–20, 2020. c⃝2020 Association for Computational Linguistics 2572 Towards Reasonably-Sized Character-Level Transformer NMT by Finetuning Subword Systems Jindˇrich Libovick´y and Alexander Fraser Center for Information and Speech Processing Ludwig Maximilian University of Munich Munich, Germany {libovicky, fraser}@cis.lmu.de Abstract Applying the Transformer architecture on the character level usually requires very deep ar- chitectures that are difficult and slow to train. These problems can be partially overcome by incorporating a segmentation into tokens in the model. We show that by initially train- ing a subword model and then finetuning it on characters, we can obtain a neural machine translation model that works at the charac- ter level without requiring token segmentation. We use only the vanilla 6-layer Transformer Base architecture. Our character-level models better capture morphological phenomena and show more robustness to noise at the expense of somewhat worse overall translation quality. Our study is a significant step towards high- performance and easy to train character-based models that are not extremely large. 1 Introduction State-of-the-art neural machine translation (NMT) models operate almost end-to-end except for input and output text segmentation. The segmentation is done by first employing rule-based tokenization and then splitting into subword units using statis- tical heuristics such as byte-pair encoding (BPE; Sennrich et al., 2016) or SentencePiece (Kudo and Richardson, 2018). Recurrent sequence-to-sequence (S2S) models can learn translation end-to-end (at the character level) without changes in the architecture (Cherry et al., 2018), given sufficient model depth. Training character-level Transformer S2S models (Vaswani et al., 2017) is more complicated because the self- attention size is quadratic in the sequence length. In this paper, we empirically evaluate Trans- former S2S models. We observe that training a character-level model directly from random initial- ization suffers from instabilities, often preventing it from converging. Instead, we propose finetun- ing subword-based models to get a model without explicit segmentation. Our character-level models show slightly worse translation quality, but have better robustness towards input noise and better capture morphological phenomena. Our approach is important because previous approaches have re- lied on very large transformers, which are out of reach for much of the research community. 2 Related Work Character-level decoding seemed to be rela- tively easy with recurrent S2S models (Chung et al., 2016). But early attempts at achieving segmentation-free NMT with recurrent networks used input hidden states covering a constant char- acter span (Lee et al., 2017). Cherry et al. (2018) showed that with a sufficiently deep recurrent model, no changes in the model are necessary, and they can still reach translation quality that is on par with subword models. Luong and Manning (2016) and Ataman et al. (2019) can leverage character- level information but they require tokenized text as an input and only have access to the character-level embeddings of predefined tokens. Training character-level transformers is more challenging. Choe et al. (2019) successfully trained a character-level left-to-right Transformer language model that performs on par with a subword-level model. However, …

Content truncated to 500 words