Predicting Target Language CCG Supertags Improves Neural Machine Translation

Proceedings of the Conference on Machine Translation (WMT), Volume 1: Research Papers, pages 68–79 Copenhagen, Denmark, September 711, 2017. c⃝2017 Association for Computational Linguistics Predicting Target Language CCG Supertags Improves Neural Machine Translation Maria N˘adejde1 and Siva Reddy1 and Rico Sennrich1 and Tomasz Dwojak1,2 Marcin Junczys-Dowmunt2 and Philipp Koehn3 and Alexandra Birch1 1School of Informatics, University of Edinburgh 2Adam Mickiewicz University 3Dep. of Computer Science, Johns Hopkins University {m.nadejde,siva.reddy, rico.sennrich, a.birch}@ed.ac.uk {t.dwojak,junczys}@amu.edu.pl, phi@jhu.edu Abstract Neural machine translation (NMT) mod- els are able to partially learn syntactic in- formation from sequential lexical informa- tion. Still, some complex syntactic phe- nomena such as prepositional phrase at- tachment are poorly modeled. This work aims to answer two questions: 1) Does explicitly modeling target language syntax help NMT? 2) Is tight integration of words and syntax better than multitask training? We introduce syntactic information in the form of CCG supertags in the decoder, by interleaving the target supertags with the word sequence. Our results on WMT data show that explicitly modeling target- syntax improves machine translation qual- ity for German→English, a high-resource pair, and for Romanian→English, a low- resource pair and also several syntactic phenomena including prepositional phrase attachment. Furthermore, a tight cou- pling of words and syntax improves trans- lation quality more than multitask training. By combining target-syntax with adding source-side dependency labels in the em- bedding layer, we obtain a total improve- ment of 0.9 BLEU for German→English and 1.2 BLEU for Romanian→English. 1 Introduction Sequence-to-sequence neural machine translation (NMT) models (Sutskever et al., 2014; Cho et al., 2014b; Bahdanau et al., 2015) are state-of-the-art on a multitude of language-pairs (Sennrich et al., 2016a; Junczys-Dowmunt et al., 2016). Part of the appeal of neural models is that they can learn to implicitly model phenomena which underlie high quality output, and some syntax is indeed cap- tured by these models. In a detailed analysis, Bentivogli et al. (2016) show that NMT signifi- cantly improves over phrase-based SMT, in par- ticular with respect to morphology and word or- der, but that results can still be improved for longer sentences and complex syntactic phenomena such as prepositional phrase (PP) attachment. Another study by Shi et al. (2016) shows that the encoder layer of NMT partially learns syntactic informa- tion about the source language, however complex syntactic phenomena such as coordination or PP attachment are poorly modeled. Recent work which incorporates additional source-side linguistic information in NMT mod- els (Luong et al., 2016; Sennrich and Haddow, 2016) show that even though neural models have strong learning capabilities, explicit features can still improve translation quality. In this work, we examine the benefit of incorporating global syn- tactic information on the target-side. We also ad- dress the question of how best to incorporate this information. For language pairs where syntac- tic resources are available on both the source and target-side, we show that approaches to incorpo- rate source syntax and target syntax are comple- mentary. We propose a method for tightly coupling words and syntax by interleaving the …

Content truncated to 500 words