MTAG: Modal-Temporal Attention Graph for Unaligned Human Multimodal Language Sequences

Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 74–85 June 6–11, 2021. ©2021 Association for Computational Linguistics 74 Probing Word Translations in the Transformer and Trading Decoder for Encoder Layers Hongfei Xu1,2 Josef van Genabith1,2 Qiuhui Liu3∗ Deyi Xiong4 1Saarland University / Saarland, Germany 2German Research Center for Artificial Intelligence / Saarland, Germany 3China Mobile Online Services / Henan, China 4Tianjin University / Tianjin, China hfxunlp@foxmail.com, Josef.Van_Genabith@dfki.de, liuqhano@foxmail.com, dyxiong@tju.edu.cn Abstract Due to its effectiveness and performance, the Transformer translation model has attracted wide attention, most recently in terms of probing-based approaches. Previous work fo- cuses on using or probing source linguistic features in the encoder. To date, the way word translation evolves in Transformer lay- ers has not yet been investigated. Naively, one might assume that encoder layers capture source information while decoder layers trans- late. In this work, we show that this is not quite the case: translation already happens progres- sively in encoder layers and even in the input embeddings. More surprisingly, we find that some of the lower decoder layers do not ac- tually do that much decoding. We show all of this in terms of a probing approach where we project representations of the layer ana- lyzed to the final trained and frozen classifier level of the Transformer decoder to measure word translation accuracy. Our findings moti- vate and explain a Transformer configuration change: if translation already happens in the encoder layers, perhaps we can increase the number of encoder layers, while decreasing the number of decoder layers, boosting decod- ing speed, without loss in translation quality? Our experiments show that this is indeed the case: we can increase speed by up to a fac- tor 2.3 with small gains in translation qual- ity, while an 18-4 deep encoder configuration boosts translation quality by +1.42 BLEU (En- De) at a speed-up of 1.4. 1 Introduction Neural Machine Translation (NMT) has achieved great success in the last few years. The popular Transformer (Vaswani et al., 2017) model, which outperforms previous RNN/CNN based transla- tion models (Bahdanau et al., 2014; Gehring et al., 2017), is based on multi-layer self-attention net- works and can be parallelized effectively. ∗ Corresponding author. Recently, a wide range of studies related to the Transformer have been conducted. For example, Bisazza and Tump (2018) perform a fine-grained analysis of how various source-side morphological features are captured at different levels of an NMT encoder. Surprisingly, they do not find any corre- lation between the accuracy of source morphology encoding and translation quality. Morphological features are only captured in context and only to the extent that they are directly transferable to target words. Voita et al. (2019a) study how information flows across Transformer layers and find that rep- resentations differ significantly depending on the objectives (machine translation, standard left-to- right language models and masked language mod- eling). Tang et al. (2019) find that encoder hidden states outperform word embeddings significantly in word sense disambiguation. …

Content truncated to 500 words