Widening the Representation Bottleneck in Neural Machine Translation with Lexical Shortcuts

Proceedings of the Fourth Conference on Machine Translation (WMT), Volume 1: Research Papers, pages 102–115 Florence, Italy, August 1-2, 2019. c⃝2019 Association for Computational Linguistics 102 Widening the Representation Bottleneck in Neural Machine Translation with Lexical Shortcuts Denis Emelin1, Ivan Titov1, 2, and Rico Sennrich1, 3 1University of Edinburgh, Scotland 2University of Amsterdam, Netherlands 3University of Zurich, Switzerland D.Emelin@sms.ed.ac.uk ititov@inf.ed.ac.uk rico.sennrich@ed.ac.uk Abstract The transformer is a state-of-the-art neural translation model that uses attention to iter- atively refine lexical representations with in- formation drawn from the surrounding con- text. Lexical features are fed into the first layer and propagated through a deep network of hid- den layers. We argue that the need to rep- resent and propagate lexical features in each layer limits the model’s capacity for learn- ing and representing other information rele- vant to the task. To alleviate this bottleneck, we introduce gated shortcut connections be- tween the embedding layer and each subse- quent layer within the encoder and decoder. This enables the model to access relevant lex- ical content dynamically, without expending limited resources on storing it within inter- mediate states. We show that the proposed modification yields consistent improvements over a baseline transformer on standard WMT translation tasks in 5 translation directions (0.9 BLEU on average) and reduces the amount of lexical information passed along the hid- den layers. We furthermore evaluate different ways to integrate lexical connections into the transformer architecture and present ablation experiments exploring the effect of proposed shortcuts on model behavior.1 1 Introduction Since it was first proposed, the transformer model (Vaswani et al., 2017) has quickly established it- self as a popular choice for neural machine trans- lation, where it has been found to deliver state-of- the-art results on various translation tasks (Bojar et al., 2018). Its success can be attributed to the model’s high parallelizability allowing for signifi- cantly faster training compared to recurrent neural 1Our code is publicly available to aid the reproduction of the reported results: https://github.com/demelin/ transformer_lexical_shortcuts networks (Chen et al., 2018), superior ability to perform lexical disambiguation, and capacity for capturing long-distance dependencies on par with existing alternatives (Tang et al., 2018). Recently, several studies have investigated the nature of features encoded within individual layers of neural translation models (Belinkov et al., 2017, 2018). One central finding reported in this body of work is that, in recurrent architectures, differ- ent layers prioritize different information types. As such, lower layers appear to predominantly perform morphological and syntactic processing, whereas semantic features reach their highest con- centration towards the top of the layer stack. One necessary consequence of this distributed learn- ing is that different types of information encoded within input representations received by the trans- lation model have to be transported to the layers specialized in exploiting them. Within the transformer encoder and decoder alike, information exchange proceeds in a strictly sequential manner, whereby each layer attends over the output of the immediately preceding layer, complemented by a shallow residual con- nection. For input features to be successfully …

Content truncated to 500 words