Beyond Weight Tying: Learning Joint Input-Output Embeddings for Neural Machine Translation

Proceedings of the Third Conference on Machine Translation (WMT), Volume 1: Research Papers, pages 73–83 Belgium, Brussels, October 31 - Novermber 1, 2018. c⃝2018 Association for Computational Linguistics https://doi.org/10.18653/v1/W18-64008 Beyond Weight Tying: Learning Joint Input-Output Embeddings for Neural Machine Translation Nikolaos Pappas† Lesly Miculicich Werlen†♦ James Henderson† †Idiap Research Institute, Martigny, Switzerland ♦ ´Ecole Polytechnique F´ed´erale de Lausanne (EPFL), Switzerland {npappas,lmiculicich,jhenderson}@idiap.ch Abstract Tying the weights of the target word em- beddings with the target word classifiers of neural machine translation models leads to faster training and often to better translation quality. Given the success of this parameter sharing, we investigate other forms of shar- ing in between no sharing and hard equal- ity of parameters. In particular, we pro- pose a structure-aware output layer which captures the semantic structure of the output space of words within a joint input-output em- bedding. The model is a generalized form of weight tying which shares parameters but allows learning a more flexible relationship with input word embeddings and allows the effective capacity of the output layer to be controlled. In addition, the model shares weights across output classifiers and transla- tion contexts which allows it to better lever- age prior knowledge about them. Our eval- uation on English-to-Finnish and English-to- German datasets shows the effectiveness of the method against strong encoder-decoder base- lines trained with or without weight tying. 1 Introduction Neural machine translation (NMT) predicts the target sentence one word at a time, and thus mod- els the task as a sequence classification problem where the classes correspond to words. Typi- cally, words are treated as categorical variables which lack description and semantics. This makes training speed and parametrization dependent on the size of the target vocabulary (Mikolov et al., 2013). Previous studies overcome this problem by truncating the vocabulary to limit its size and mapping out-of-vocabulary words to a single “un- known” token. Other approaches attempt to use a limited number of frequent words plus sub-word units (Sennrich et al., 2016), the combination of which can cover the full vocabulary, or to perform character-level modeling (Chung et al., 2016; Lee et al., 2017; Costa-juss`a and Fonollosa, 2016; Ling et al., 2015); with the former being the most effec- tive between the two. The idea behind these al- ternatives is to overcome the vocabulary size issue by modeling the morphology of rare words. One limitation, however, is that semantic information of words or sub-word units learned by the input embedding are not considered when learning to predict output words. Hence, they rely on a large amount of examples per class to learn proper word or sub-word unit output classifiers. One way to consider information learned by in- put embeddings, albeit restrictively, is with weight tying i.e. sharing the parameters of the input em- beddings with those of the output classifiers (Press and Wolf, 2017; Inan et al., 2016) which is effec- tive for language modeling and machine transla- tion (Sennrich et al., 2017; Klein et al., 2017). De- spite its usefulness, we …

Content truncated to 500 words