Extending hybrid word-character neural machine translation with multi-task learning of morphological analysis

Proceedings of the Conference on Machine Translation (WMT), Volume 2: Shared Task Papers, pages 296–302 Copenhagen, Denmark, September 711, 2017. c⃝2017 Association for Computational Linguistics Extending hybrid word-character neural machine translation with multi-task learning of morphological analysis Stig-Arne Grönroos and Sami Virpioja and Mikko Kurimo stig-arne.gronroos@aalto.fi Department of Signal Processing and Acoustics, Aalto University, Finland Abstract This article describes the Aalto University entry to the English-to-Finnish news trans- lation shared task in WMT 2017. Our sys- tem is an open vocabulary neural machine translation (NMT) system, adapted to the needs of a morphologically complex target language. The main contributions of this paper are 1) implicitly incorporating mor- phological information to NMT through multi-task learning, 2) adding an attention mechanism to the character-level decoder, combined with character segmentation of names, and 3) a new overattending penalty to beam search. 1 Introduction The rich inflection, derivation and compounding in synthetic languages can result in very large vocab- ularies. In statistical machine translation (SMT) large vocabularies cause sparsity issues. While continuous space representations make neural ma- chine translation (NMT) more robust towards such sparsity, it suffers from a different set of prob- lems related to large vocabularies. A large vo- cabulary bloats memory and computation require- ments, while still leaving the problem of out-of- vocabulary words unsolved. Subword vocabularies have been proposed as a solution. While the benefits of using subwords in SMT have been at best moderate (Virpioja et al., 2007; Fishel and Kirik, 2010; Grönroos et al., 2015), subword decoding has become popular in NMT (Sennrich et al., 2015). A subword vocabu- lary of a moderate size ensures full coverage of an open vocabulary. The downside is an increase in the length of the input and output sequences. Long sequences cause a large increase in computation time, especially for architectures using the atten- tion mechanism. An alternative approach is the hybrid word- character decoder presented by Luong and Man- ning (2016). In the hybrid decoder, a word level decoder outputs frequent words as they are, while replacing infrequent words with a special <UNK> symbol. A second character-level decoder then ex- pands these <UNK> symbols into surface forms. In addition to providing moderate length of in- put and output sequences together with an open vocabulary, the hybrid word-character decoder makes it simple to use labels based on the level of words, provided for example by morphological analyzers and parsers. In SMT, such tools are typ- ically used via factored translation models (Koehn and Hoang, 2007). Factored translation has also been successfully applied in NMT. For example, Sennrich and Haddow (2016) augment the source words with four additional factors: PoS, lemma, dependency label and subwords. García-Martínez et al. (2016) use a decomposed generation process, in which they first output lemma, PoS, tense, per- son, gender, and number, from which the surface form is generated using a rule-based morphologi- cal analyzer. Neural machine translation provides another way to utilize external annotations, multi-task learning (MTL). MTL is a well established ma- chine learning approach that …

Content truncated to 500 words