Neural Machine Translation for Bilingually Scarce Scenarios: a Deep Multi-Task Learning Approach

Proceedings of NAACL-HLT 2018, pages 1356–1365 New Orleans, Louisiana, June 1 - 6, 2018. c⃝2018 Association for Computational Linguistics Neural Machine Translation for Bilingually Scarce Scenarios: A Deep Multi-task Learning Approach Poorya Zaremoodi Gholamreza Haffari Faculty of Information Technology, Monash University, Australia first.last@monash.edu Abstract Neural machine translation requires large amounts of parallel training text to learn a reasonable-quality translation model. This is particularly inconvenient for language pairs for which enough parallel text is not available. In this paper, we use monolingual linguistic re- sources in the source side to address this chal- lenging problem based on a multi-task learn- ing approach. More specifically, we scaffold the machine translation task on auxiliary tasks including semantic parsing, syntactic parsing, and named-entity recognition. This effectively injects semantic and/or syntactic knowledge into the translation model, which would other- wise require a large amount of training bitext. We empirically evaluate and show the effec- tiveness of our multi-task learning approach on three translation tasks: English-to-French, English-to-Farsi, and English-to-Vietnamese. 1 Introduction Neural Machine Translation (NMT) with atten- tional encoder-decoder architectures (Luong et al., 2015; Bahdanau et al., 2015) has revolutionised machine translation, and achieved state-of-the-art for several language pairs. However, NMT is no- torious for its need for large amounts of bilin- gual data (Koehn and Knowles, 2017) to achieve reasonable translation quality. Leveraging exist- ing monolingual resources is a potential approach for compensating this requirement in bilingually scarce scenarios. Ideally, semantic and syntac- tic knowledge learned from existing linguistic re- sources provides NMT with proper inductive bi- ases, leading to increased generalisation and better translation quality. Multi-task learning (MTL) is an effective ap- proach to inject knowledge into a task, which is learned from other related tasks. Various recent works have attempted to improve NMT with an MTL approach (Peng et al., 2017; Liu et al., 2017; Zhang and Zong, 2016); however, they either do not make use of curated linguistic resources (Domhan and Hieber, 2017; Zhang and Zong, 2016), or their MTL architectures are restrictive yielding mediocre improvements (Niehues and Cho, 2017). The current research leaves open how to best leverage curated linguistic resources in a suitable MTL framework to improve NMT. In this paper, we make use of curated monolin- gual linguistic resources in the source side to im- prove NMT in bilingually scarce scenarios. More specifically, we scaffold the machine translation task on auxiliary tasks including semantic pars- ing, syntactic parsing, and named-entity recogni- tion. This is achieved by casting the auxiliary tasks as sequence-to-sequence (SEQ2SEQ) trans- duction tasks, and tie the parameters of their en- coders and/or decoders with those of the main translation task. Our MTL architectures makes use of deep stacked encoders and decoders, where the parameters of the top layers are shared across the tasks. We further make use of adversarial training to prevent contamination of common knowledge with task-specific information. We present empirical results on translating from English into French, Vietnamese, and Farsi; three target languages with varying degree of diver- gence compared to English. Our extensive …

Content truncated to 500 words