Modeling Target-side Inflection in Placeholder Translation

Learning Curricula for Multilingual Neural Machine Translation Training Gaurav Kumar gkumar@cs.jhu.edu Philipp Koehn phi@jhu.edu Sanjeev Khudanpur khudanpur@jhu.edu CLSP, Johns Hopkins University, Baltimore, 21218, USA Abstract Low-resource Multilingual Neural Machine Translation (MNMT) is typically tasked with improving the translation performance on one or more language pairs with the aid of high- resource language pairs. In this paper, we propose two simple search based curricula – orderings of the multilingual training data – which help improve translation performance in conjunction with existing techniques such as fine-tuning. Additionally, we attempt to learn a curriculum for MNMT from scratch jointly with the training of the translation system using contextual multi-arm bandits. We show on the FLORES low-resource translation dataset that these learned curricula can provide better starting points for fine tuning and improve overall performance of the translation system. 1 Introduction Curriculum learning (Bengio et al., 2009; Elman, 1993; Rohde and Plaut, 1994) hypothesizes that presenting training samples in a meaningful order to machine learners during training may help improve model quality and convergence speed. In the field of Neural Machine Translation (NMT) most curricula are hand designed e.g., fine-tuning (Luong and Manning, 2015; Freitag and Al-Onaizan, 2016) and data selection (Moore and Lewis, 2010; Axelrod et al., 2011; Duh et al., 2013; Durrani et al., 2016). Another common curriculum is one based on ordering samples from easy to hard using linguistic features and auxiliary model scores (Zhang et al., 2018, 2019) but these are hard to tune, relying to extensive trial and error to find the right hyperparameters. Attempts to learn a curriculum jointly with the NMT training setup (Kumar et al., 2019) can suffer from observation sparsity, where a single training run does not provide enough training samples for an external agent to learn a good curriculum policy. Our NMT task of choice in this paper is low-resource multi-lingual NMT (MNMT). While standard NMT systems typically deal with a language pair, the source and the target, an MNMT model may have multiple languages as source and/or target. Most large-scale MNMT models are trained using some form of model parameter sharing (Johnson et al., 2017; Aharoni et al., 2019; Arivazhagan et al., 2019; Bapna and Firat, 2019). The notion of how input data should be presented to the MNMT system during training only finds prominence in the case of low-resource MNMT. A typical low-resource task will try to leverage a high-resource language pair to aid the training of an NMT system for a low-resource (very small or no parallel data available) and related language-pair of interest. Typical approaches for low resource MNMT involve pivoting and zero-shot training (Lakew et al., 2018; Johnson et al., 2017) and transfer learning via fine-tuning (Zoph et al., 2016; Dabre et al., 2019). Finn et al. (2017) attempt to meta-learn parameter initialization for child models using trained-high resource parent models for this task. Proceedings of the 18th Biennial Machine Translation Summit Virtual USA, August 16 - 20, 2021, Volume 1: MT Research Track Page 1 Figure 1: The …

Content truncated to 500 words