Character-Level Translation with Self-attention

Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 1591–1604 July 5 - 10, 2020. c⃝2020 Association for Computational Linguistics 1591 Character-Level Translation with Self-attention Yingqiang Gao†‡, Nikola I. Nikolov‡, Yuhuang Hu‡, Richard H.R. Hahnloser‡ †Department of Informatics, Technical University of Munich ‡Institute of Neuroinformatics, University of Zurich and ETH Zurich yingqiang.gao@in.tum.de {niniko, yuhuang.hu, rich}@ini.ethz.ch Abstract We explore the suitability of self-attention models for character-level neural machine translation. We test the standard transformer model, as well as a novel variant in which the encoder block combines information from nearby characters using convolutions. We perform extensive experiments on WMT and UN datasets, testing both bilingual and mul- tilingual translation to English using up to three input languages (French, Spanish, and Chinese). Our transformer variant consis- tently outperforms the standard transformer at the character-level and converges faster while learning more robust character-level align- ments.1 1 Introduction Most existing Neural Machine Translation (NMT) models operate on the word or subword-level, which tends to make these models memory ineffi- cient because of large vocabulary sizes. Character- level models (Lee et al., 2017; Cherry et al., 2018) instead work directly on raw characters, result- ing in a more compact language representation, while mitigating out-of-vocabulary (OOV) prob- lems (Luong and Manning, 2016). Character- level models are also very suitable for multilingual translation since multiple languages can be mod- eled using the same character vocabulary. Multi- lingual training can lead to improvements in the overall performance without an increase in model complexity (Lee et al., 2017), while also circum- venting the need to train separate models for each language pair. Models based on self-attention have achieved excellent performance on a number of tasks, in- cluding machine translation (Vaswani et al., 2017) and representation learning (Devlin et al., 2019; 1Code available at https://github.com/ CharizardAcademy/convtransformer Multi-Head Attention Add & Norm Feed Forward Add & Norm Conv 1D width 3 Conv 1D width 5 Conv 1D width 7 concat. Conv 1D width 3 Parallel Conv 1D layer Parallel Conv 1D Layer Multi-Head Attention Add & Norm Feed Forward Add & Norm a) b) Figure 1: A comparison of the encoder blocks in the standard transformer (a) and our novel modification, the convtransformer (b), which uses 1D convolutions to facilitate character interactions. Yang et al., 2019). Despite the success of these models, their suitability for character-level trans- lation remains largely unexplored, with most ef- forts having focused on recurrent models (e.g., Lee et al. (2017); Cherry et al. (2018)). In this work, we perform an in-depth investiga- tion of the suitability of self-attention models for character-level translation. We consider two mod- els: the standard transformer from Vaswani et al. (2017) and a novel variant that we call the con- vtransformer (Figure 1, Section 3). The convtrans- former uses convolutions to facilitate interactions among nearby character representations. We evaluate these models on both bilingual and multilingual translation to English, using up to three input languages: French (FR), Spanish (ES), and Chinese (ZH). We compare the perfor- mance when …

Content truncated to 500 words