JASS: Japanese-specific Sequence to Sequence Pre-training for Neural Machine Translation

Proceedings of the 12th Conference on Language Resources and Evaluation (LREC 2020), pages 3683–3691 Marseille, 11–16 May 2020 c⃝ European Language Resources Association (ELRA), licensed under CC-BY-NC 3683 JASS: Japanese-specific Sequence to Sequence Pre-training for Neural Machine Translation Zhuoyuan Mao†, Fabien Cromieres†⋆, Raj Dabre‡⋆, Haiyue Song†, Sadao Kurohashi† †Kyoto University ‡National Institute of Information and Communications Technology {zhuoyuanmao, fabien, song, kuro}@nlp.ist.i.kyoto-u.ac.jp, raj.dabre@nict.go.jp Abstract Neural machine translation (NMT) needs large parallel corpora for state-of-the-art translation quality. Low-resource NMT is typically addressed by transfer learning which leverages large monolingual or parallel corpora for pre-training. Monolingual pre-training approaches such as MASS (MAsked Sequence to Sequence) are extremely effective in boosting NMT quality for languages with small parallel corpora. However, they do not account for linguistic information obtained using syntactic analyzers which is known to be invaluable for several Natural Language Processing (NLP) tasks. To this end, we propose JASS, Japanese-specific Sequence to Sequence, as a novel pre-training alternative to MASS for NMT involving Japanese as the source or target language. JASS is joint BMASS (Bunsetsu MASS) and BRSS (Bunsetsu Reordering Sequence to Sequence) pre-training which focuses on Japanese linguistic units called bunsetsus. In our experiments on ASPEC Japanese–English and News Commentary Japanese–Russian translation we show that JASS can give results that are competitive with if not better than those given by MASS. Furthermore, we show for the first time that joint MASS and JASS pre-training gives results that significantly surpass the individual methods indicating their complementary na- ture. We will release our code, pre-trained models and bunsetsu annotated data as resources for researchers to use in their own NLP tasks. Keywords: pre-training, neural machine translation, bunsetsu, low resource 1. Introduction Encoder-decoder based neural machine translation (NMT) (Sutskever et al., 2014; Bahdanau et al., 2015), and in par- ticular, the Transformer model (Vaswani et al., 2017) have led to a large jump in the quality of automatic transla- tion over previous approaches such as Statistical Machine Translation (Koehn, 2004). One of the drawbacks of NMT is that it requires large parallel corpora for training robust and high quality translation models. This strongly limits its usefulness for many language pairs and domains for which no such large corpora exist. The most popular way to solve this issue is to leverage monolingual corpora, which are much easier to obtain (as compared to parallel corpora) for most languages and do- mains. This can be done either by backtranslation (Sen- nrich et al., 2016a; Hoang et al., 2018; Edunov et al., 2018) or by pre-training. Pre-training consists in initializing some or all of the parameters of the model through tasks that only require monolingual data. One can pre-train the word em- beddings of the model (Qi et al., 2018) or the encoder and decoders (Zoph et al., 2016). Pre-training has recently be- come the focus of much research after the success of meth- ods such as BERT (Devlin et al., 2019), ELMO (Peters et al., 2018) or GPT (Radford, 2018) in many NLP tasks. However, these methods were …

Content truncated to 500 words