Input Combination Strategies for Multi-Source Transformer Decoder

Proceedings of the Third Conference on Machine Translation (WMT), Volume 1: Research Papers, pages 253–260 Belgium, Brussels, October 31 - Novermber 1, 2018. c⃝2018 Association for Computational Linguistics https://doi.org/10.18653/v1/W18-64026 Input Combination Strategies for Multi-Source Transformer Decoder Jindˇrich Libovick´y and Jindˇrich Helcl and David Mareˇcek Charles University, Faculty of Mathematics and Physics Institute of Formal and Applied Linguistics Malostransk´e n´amˇest´ı 25, 118 00 Prague, Czech Republic {libovicky, helcl, marecek}@ufal.mff.cuni.cz Abstract In multi-source sequence-to-sequence tasks, the attention mechanism can be modeled in several ways. This topic has been thoroughly studied on recurrent architectures. In this paper, we extend the previous work to the encoder-decoder attention in the Transformer architecture. We propose four different in- put combination strategies for the encoder- decoder attention: serial, parallel, flat, and hi- erarchical. We evaluate our methods on tasks of multimodal translation and translation with multiple source languages. The experiments show that the models are able to use multiple sources and improve over single source base- lines. 1 Introduction The Transformer model (Vaswani et al., 2017) re- cently demonstrated superior performance in neu- ral machine translation (NMT) and other sequence generation tasks such as text summarization or im- age captioning (Kaiser et al., 2017). However, all of these setups consider only a single input to the decoder part of the model. In the Transformer architecture, the represen- tation of the source sequence is supplied to the decoder through the encoder-decoder attention. This attention sub-layer is applied between the self-attention and feed-forward sub-layers in each Transformer layer. Such arrangement leaves many options for the incorporation of multiple encoders. So far, attention in sequence-to-sequence learn- ing with multiple source sequences was mostly studied in the context of recurrent neural networks (RNNs). Libovick´y and Helcl (2017) explicitly capture the distribution over multiple inputs by projecting the input representations to a shared vector space and either computing the attention over all hidden states at once, or hierarchically, us- ing another level of attention applied on the con- text vectors. Zoph and Knight (2016) employ a gating mechanism for combining the context vec- tors. Voita et al. (2018) adapted the gating mech- anism for use within the Transformer model for context-aware MT. The other aproaches are how- ever not directly usable in the Transformer model. We propose a number of strategies of com- bining the different sources in the Transformer model. Some of the strategies described in this work are an adaptation of the strategies previously used with recurrent neural networks (Libovick´y and Helcl, 2017), whereas the rest of them is a novel contribution devised for the Transformer ar- chitecture. We test these strategies on multimodal machine translation (MMT) and multi-source ma- chine translation (MSMT) tasks. This paper is organized as follows. In Sec- tion 2, we briefly describe the decoder part of the Transformer model. We propose a number of input combination strategies for the multi-source Transformer model in Section 3. Section 4 de- scribes the experiments we performed, and Sec- tion 5 shows the results of quantitative evaluation. An overview …

Content truncated to 500 words