Factored Neural Machine Translation Architectures

Factored Neural Machine Translation Architectures Mercedes Garc´ıa-Mart´ınez, Lo¨ıc Barrault, Fethi Bougares LIUM, University of Le Mans, France firstname.lastname@univ-lemans.fr Abstract In this paper we investigate the potential of the neural ma- chine translation (NMT) when taking into consideration the linguistic aspect of target language. From this standpoint, the NMT approach with attention mechanism [1] is extended in order to produce several linguistically derived outputs. We train our model to simultaneously output the lemma and its corresponding factors (e.g. part-of-speech, gender, number). The word level translation is built with a mapping function using a priori linguistic information. Compared to the stan- dard NMT system, factored architecture increases signifi- cantly the vocabulary coverage while decreasing the num- ber of unknown words. With its richer architecture, the Fac- tored NMT approach allows us to implement several training setup that will be discussed in detail along this paper. On the IWSLT’15 English-to-French task, FNMT model outper- forms NMT model in terms of BLEU score. A qualitative analysis of the output on a set of test sentences shows the effectiveness of the FNMT model. 1. Introduction Neural Machine Translation (NMT) approach has been fur- ther developed in the last years [1, 2]. In contrast to the traditional phrased-based statistical machine translation [3] that represents and translates the input sentence with a set of phrases, NMT uses the sequence to sequence learning ar- chitecture and the whole input sentence is considered as one unit for translation [4]. Recently, NMT is gaining more and more interest and showing better accuracy than phrase-based system translating several language pairs. In spite of these recent improvements, the NMT systems still have some re- strictions and difficulties to translate. One of them is the high computational of the softmax function which requires to nor- malize all the output vocabulary size. In order to solve this issue, a standard technique is to define a short-list containing only the most frequent words. This has the disadvantage of increasing the number of un- known words. In [5], authors propose to carefully organise the batches so that only a subset K of the target vocabulary is possibly generated at training time. This allows the system to train a model with much larger target vocabulary without substan- tially increasing the computational complexity. Another al- ternative is proposed by [6] where a structured output layer (SOUL) is defined to handle the words not appearing in the shortlist. This allows the system to always apply the softmax normalization on a layer with reduced size. Recently, some works have used subword units level instead of word-level for translation. In [7], the rare and unknown words are encoded as subword units with the Byte Pair Encoding (BPE) algorithm. The authors show that this method can generate words which are unseen at training time. Another lower level for translation is the character-level NMT, which has been presented in several works [8, 9, 10] and showed promising results. Factored Neural Machine Translation (FNMT) ap- proach handles the output vocabulary size problem using fac- …

Content truncated to 500 words