An Analysis of Encoder Representations in Transformer-Based Machine Translation

Proceedings of the 2018 EMNLP Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP, pages 287–297 Brussels, Belgium, November 1, 2018. c⃝2018 Association for Computational Linguistics 287 An Analysis of Encoder Representations in Transformer-Based Machine Translation Alessandro Raganato and J¨org Tiedemann Department of Digital Humanities University of Helsinki {alessandro.raganato,jorg.tiedemann}@helsinki.fi Abstract The attention mechanism is a successful tech- nique in modern NLP, especially in tasks like machine translation. The recently proposed network architecture of the Transformer is based entirely on attention mechanisms and achieves new state of the art results in neu- ral machine translation, outperforming other sequence-to-sequence models. However, so far not much is known about the internal prop- erties of the model and the representations it learns to achieve that performance. To study this question, we investigate the information that is learned by the attention mechanism in Transformer models with different transla- tion quality. We assess the representations of the encoder by extracting dependency rela- tions based on self-attention weights, we per- form four probing tasks to study the amount of syntactic and semantic captured information and we also test attention in a transfer learn- ing scenario. Our analysis sheds light on the relative strengths and weaknesses of the vari- ous encoder representations. We observe that specific attention heads mark syntactic depen- dency relations and we can also confirm that lower layers tend to learn more about syntax while higher layers tend to encode more se- mantics. 1 Introduction Machine translation (MT) is one of the promi- nent tasks in Natural Language Processing, tack- led in several ways (Bojar et al., 2017). Neural MT (NMT) has become the de-facto standard with a performance that clearly outperforms the alter- native approach of Statistical Machine Transla- tion (Luong et al., 2015b; Bojar et al., 2016; Ben- tivogli et al., 2016). NMT also improves training procedures due to the end-to-end fashion without tedious feature engineering and complex setups. During recent years, a lot of research has been done on NMT, designing new architectures, start- ing from the plain sequence-to-sequence model (Sutskever et al., 2014; Cho et al., 2014), to an im- proved version featuring an attention mechanism (Bahdanau et al., 2015; Luong et al., 2015a), to models that only use attention instead of recurrent layers (Vaswani et al., 2017) and models that ap- ply convolution networks (Gehring et al., 2017a,b). Among the different architectures, the Transformer (Vaswani et al., 2017) has emerged as the dominant NMT paradigm.1 Relying only on attention mech- anisms, the model is fast, highly accurate and has been proven to outperform the widely used recur- rent networks with attention and ensembling (Wu et al., 2016) by more than 2 BLEU points. Im- proved translation quality is typically related to bet- ter representation of structural information. While other approaches make use of external information to improve the internal representation of NMT mod- els (Arthur et al., 2016; Niehues and Cho, 2017; Alkhouli and Ney, 2017), the Transformer seems to be able to encode a lot of structural informa- tion …

Content truncated to 500 words