Understanding Pure Character-Based Neural Machine Translation: The Case of Translating Finnish into English

Proceedings of the 28th International Conference on Computational Linguistics, pages 4251–4262 Barcelona, Spain (Online), December 8-13, 2020 4251 Understanding Pure Character-Based Neural Machine Translation: The Case of Translating Finnish into English Gongbo Tang1 Rico Sennrich2,3 Joakim Nivre1 1Department of Linguistics and Philology, Uppsala University 2Institute of Computational Linguistics, University of Zurich 3School of Informatics, University of Edinburgh firstname.lastname@{lingfil.uu.se, ed.ac.uk} Abstract Recent work has shown that deeper character-based neural machine translation (NMT) models can outperform subword-based models. However, it is still unclear what makes deeper character-based models successful. In this paper, we conduct an investigation into pure character-based models in the case of translating Finnish into English, including exploring the ability to learn word senses and morphological inflections and the attention mechanism. We demonstrate that word-level information is distributed over the entire character sequence rather than over a single character, and characters at different positions play different roles in learning linguistic knowledge. In addition, character-based models need more layers to encode word senses which explains why only deeper models outperform subword-based models. The attention distribution pattern shows that separators attract a lot of attention and we explore a sparse word-level attention to enforce character hidden states to capture the full word-level information. Experimental results show that the word-level attention with a single head results in 1.2 BLEU points drop. 1 Introduction Neural machine translation (NMT) has boosted machine translation significantly in recent years (Kalch- brenner and Blunsom, 2013; Cho et al., 2014; Sutskever et al., 2014; Bahdanau et al., 2015; Luong et al., 2015; Gehring et al., 2017; Vaswani et al., 2017). However, it is still unclear how NMT models work due to the black-box nature of neural networks. Better understandings of NMT models could guide us in improving NMT systems. Currently most of the studies towards understanding NMT models only take into account subword-based (e.g. BPE-based) models. Deeper character-based (CHAR) models have been shown to perform better than BPE-based models (Cherry et al., 2018). In this paper, we try to investigate the working mechanism of CHAR models. We explore the ability of CHAR models to learn word senses and morphological inflections and the attention mechanism. Previous studies have tried to interpret and understand NMT models by interpreting attention weights (Ghader and Monz, 2017; Raganato and Tiedemann, 2018; Tang et al., 2018; Tang et al., 2019), using gradients (He et al., 2019), applying layer-wise relevance propagation (Ding et al., 2017), probing classification tasks (Belinkov et al., 2017b; Belinkov et al., 2017a; Belinkov et al., 2020; Poliak et al., 2018; Tang et al., 2019), and more intrinsic analysis (Ghader and Monz, 2019; Voita et al., 2019). However, only Belinkov et al. (2017a; Belinkov et al. (2020) have probed character-based representations. Belinkov et al. (2017a) have only explored character-aware word-level representations, while we investigate fully character-level representations, which are also studied in Belinkov et al. (2020). We apply more composition methods to explore how CHAR models learn linguistic knowledge and how attention extracts features directly from characters. Probing classification tasks (Belinkov et al., …

Content truncated to 500 words