Effective Strategies in Zero-Shot Neural Machine Translation

Effective Strategies in Zero-Shot Neural Machine Translation Thanh-Le Ha, Jan Niehues, Alexander Waibel Institute for Anthropomatics and Robotics KIT - Karlsruhe Institute of Technology, Germany firstname.lastname@kit.edu Abstract In this paper, we proposed two strategies which can be ap- plied to a multilingual neural machine translation system in order to better tackle zero-shot scenarios despite not having any parallel corpus. The experiments show that they are effective in terms of both performance and computing re- sources, especially in multilingual translation of unbalanced data in real zero-resourced condition when they alleviate the language bias problem. 1. Introduction The newly proposed neural machine translation [1] has shown the best performance in recent machine translation campaigns for several language pair. Being applied to mul- tilingual settings, neural machine translation (NMT) systems have been proved to be benefited from additional information embedded in a common semantic space across languages. However, in the extreme cases where no parallel data is available to train such system, often NMT systems suffer a bad training situation and are incapable to perform adequate translation. In this work, we point out the underlying problem of current multilingual NMT systems when dealing with zero- resource scenarios. Then we propose two simple strategies to reduce adverse impact of the problem. The strategies need little modifications in the standard NMT framework, yet they are still able to achieve better performance on zero-shot translation tasks with much less training time. 1.1. Neural Machine Translation In this section, we briefly describe the framework of Neu- ral Machine Translation as a sequence-to-sequence modeling problem following the proposed method of [1]. Given a source sentence x = (x1, .., xi, .., xI) and the corresponding target sentence y = (y1, .., yj, .., yJ), the NMT aims to directly model the translation probability of the target sequence: P(y|x) = IY j=1 P(yj|y<j, x; ✓) [1] proposed an encoder-attention-decoder framework to calculate this probability. A bidirectional recurrent encoder reads a word xi from the source sentence and produces a representation of the sen- tence in a fixed-length vector hi concatenated from those of the forward and backward directions: hi = [−! h i, − h i] −! h i = d(−! h i−1, Es • xi) (1) − h i = d( − h i+1, Es • xi) (2) where Es is the source word embedding matrix to be shared across the source words xi 2 Vx, d is the recurrent unit computing the current hidden state of the encoder based on the previous hidden state. hi is then called an annotation vector which encodes the source sentence up to the time i from both forward and backward directions. Then an attention mechanism is set up in order to choose which annotation vectors should contribute to the predicting decision of the next target word. Normally, a relevance score rel(zj−1, hi) between the previous target word and the an- notation vectors is used to calculate the context vector ci: ↵ij = exp(rel(zj−1, hi)) P i0 exp(rel(zj−1, hi0)), cj = X …

Content truncated to 500 words