An Intrinsic Nearest Neighbor Analysis of Neural Machine Translation Architectures

An Intrinsic Nearest Neighbor Analysis of Neural Machine Translation Architectures Hamidreza Ghader Informatics Institute, University of Amsterdam, The Netherlands h.ghader@uva.nl Christof Monz Informatics Institute, University of Amsterdam, The Netherlands c.monz@uva.nl Abstract Earlier approaches indirectly studied the information captured by the hidden states of recurrent and non-recurrent neural ma- chine translation models by feeding them into different classifiers. In this paper, we look at the encoder hidden states of both transformer and recurrent machine trans- lation models from the nearest neighbors perspective. We investigate to what ex- tent the nearest neighbors share informa- tion with the underlying word embeddings as well as related WordNet entries. Addi- tionally, we study the underlying syntactic structure of the nearest neighbors to shed light on the role of syntactic similarities in bringing the neighbors together. We com- pare transformer and recurrent models in a more intrinsic way in terms of capturing lexical semantics and syntactic structures, in contrast to extrinsic approaches used by previous works. In agreement with the ex- trinsic evaluations in the earlier works, our experimental results show that transform- ers are superior in capturing lexical seman- tics, but not necessarily better in capturing the underlying syntax. Additionally, we show that the backward recurrent layer in a recurrent model learns more about the se- mantics of words, whereas the forward re- current layer encodes more context. 1 Introduction Neural machine translation (NMT) has achieved state-of-the-art performance for many language c⃝ 2019 The authors. This article is licensed under a Creative Commons 4.0 licence, no derivative works, attribution, CC- BY-ND. pairs (Bahdanau et al., 2015; Luong et al., 2015b; Jean et al., 2015; Wu et al., 2016; Vaswani et al., 2017). Additionally, it is straightforward to train an NMT system in an end-to-end fashion. This has been made possible with an encoder- decoder architecture that encodes the source sen- tence into a distributed representation and then de- codes this representation into a sentence in the target language. While earlier work has investi- gated what information is captured by the atten- tion mechanism of an NMT system (Ghader and Monz, 2017), it is not exactly clear what linguistic information from the source sentence is encoded in the hidden distributed representation themselves. Recently, some attempts have been made to shed some light on the information that is being encoded in the intermediate distributed representations (Shi et al., 2016; Belinkov et al., 2017). Feeding the hidden states of the encoder of dif- ferent seq2seq systems, including multiple NMT systems, as the input to different classifiers, Shi et al. (2016) aim to show what syntactic informa- tion is encoded in the hidden states. They pro- vide evidence that syntactic information such as the voice and tense of a sentence and the part-of- speech (POS) tags of words are being learned with reasonable accuracy. They also provide evidence that more complex syntactic information such as the parse tree of a sentence is also learned, but with lower accuracy. Belinkov et al. (2017) follow the same approach as Shi et …

Content truncated to 500 words