Understanding Neural Machine Translation by Simplification: The Case of Encoder-free Models

Proceedings of Recent Advances in Natural Language Processing, pages 1186–1193, Varna, Bulgaria, Sep 2–4, 2019. https://doi.org/10.26615/978-954-452-056-4_136 1186 Understanding Neural Machine Translation by Simplification: The Case of Encoder-free Models Gongbo Tang1 Rico Sennrich2,3 Joakim Nivre1 1Department of Linguistics and Philology, Uppsala University 2School of Informatics, University of Edinburgh 3Institute of Computational Linguistics, University of Zurich firstname.lastname@{lingfil.uu.se, ed.ac.uk} Abstract In this paper, we try to understand neu- ral machine translation (NMT) via sim- plifying NMT architectures and training encoder-free NMT models. In an encoder- free model, the sums of word embeddings and positional embeddings represent the source. The decoder is a standard Trans- former or recurrent neural network that directly attends to embeddings via atten- tion mechanisms. Experimental results show (1) that the attention mechanism in encoder-free models acts as a strong fea- ture extractor, (2) that the word embed- dings in encoder-free models are compet- itive to those in conventional models, (3) that non-contextualized source represen- tations lead to a big performance drop, and (4) that encoder-free models have dif- ferent effects on alignment quality for German→English and Chinese→English. 1 Introduction Neural machine translation (NMT) (Kalchbrenner and Blunsom, 2013; Sutskever et al., 2014; Bah- danau et al., 2015; Luong et al., 2015) has emerged in the last few years and has achieved new state-of- the-art performance. However, NMT models are black boxes for humans and are hard to interpret. NMT models employ encoder-decoder architec- tures where an encoder encodes source-side sen- tences and an attentional decoder generates target- side sentences based on the outputs of the encoder. In this paper, we attempt to obtain a more inter- pretable NMT model by simplifying the encoder- decoder architecture. We train encoder-free mod- els where the sums of word embeddings and si- nusoid embeddings (Vaswani et al., 2017) repre- sent the source. The decoder is a standard Trans- former (Vaswani et al., 2017) or recurrent neural network (RNN) that attends to embeddings via at- tention mechanisms. As motivation for our architecture simplifica- tion, consider the attention mechanism1 (Bah- danau et al., 2015; Luong et al., 2015), which has been introduced to extract features from the hid- den representations in encoders dynamically. At- tention and alignment were initially used inter- changeably, but it was soon discovered that the attention mechanism can behave very differently from traditional word alignment (see Ghader and Monz, 2017; Koehn and Knowles, 2017). One rea- son for this discrepancy is that the attention mech- anism operates on representations that potentially includes information from the whole sentence due to the encoder’s recurrent or self-attentional archi- tecture. Intuitively, bypassing these encoder lay- ers and attending word embeddings directly could lead to a more alignment-like, and thus predictable and interpretable behavior of the attention model. By comparing encoder-free models with con- ventional models, we can better understand the working mechanism of NMT, figure out which components are more crucial, and learn lessons for improvement. Experimental results show that there is a significant gap between the two models. We focus on exploring what leads to …

Content truncated to 500 words