Scheduled Multi-Task Learning: From Syntax to Translation

Scheduled Multi-Task Learning: From Syntax to Translation Eliyahu Kiperwasser∗ Computer Science Department Bar-Ilan University Ramat-Gan, Israel elikip@gmail.com Miguel Ballesteros IBM Research 1101 Kitchawan Road, Route 134 Yorktown Heights, NY 10598. U.S miguel.ballesteros@ibm.com Abstract Neural encoder-decoder models of machine translation have achieved impressive results, while learning linguistic knowledge of both the source and target languages in an implicit end-to-end manner. We propose a framework in which our model begins learning syntax and translation interleaved, gradually putting more focus on translation. Using this approach, we achieve considerable improvements in terms of BLEU score on relatively large parallel cor- pus (WMT14 English to German) and a low- resource (WIT German to English) setup. 1 Introduction Neural Machine Translation (NMT) (Kalchbrenner and Blunsom, 2013; Sutskever et al., 2014; Bah- danau et al., 2014) has recently become the state- of-the-art approach to machine translation (Bojar et al., 2016). One of the main advantages of neural ap- proaches is the impressive ability of RNNs to act as feature extractors over the entire input (Kiperwasser and Goldberg, 2016), rather than focusing on local information. Neural architectures are able to extract linguistic properties from the input sentence in the form of morphology (Belinkov et al., 2017) or syn- tax (Linzen et al., 2016). Nonetheless, as shown in Dyer et al. (2016) and Dyer (2017), systems that ignore explicit linguis- tic structures are incorrectly biased and they tend to make overly strong linguistic generalizations. Pro- viding explicit linguistic information (Dyer et al., ∗ Work carried out during summer internship at IBM Re- search. 2016; Kuncoro et al., 2017; Niehues and Cho, 2017; Sennrich and Haddow, 2016; Eriguchi et al., 2017; Aharoni and Goldberg, 2017; Nadejde et al., 2017; Bastings et al., 2017; Matthews et al., 2018) has proven to be beneficial, achieving higher results in language modeling and machine translation. Multi-task learning (MTL) consists of being able to solve synergistic tasks with a single model by jointly training multiple tasks that look alike. The fi- nal dense representations of the neural architectures encode the different objectives, and they leverage the information from each task to help the others. For example, tasks like multiword expression de- tection and part-of-speech tagging have been found very useful for others like combinatory categorical grammar (CCG) parsing, chunking and super-sense tagging (Bingel and Søgaard, 2017). In order to perform accurate translations, we pro- ceed by analogy to humans. It is desirable to acquire a deep understanding of the languages; and, once this is acquired it is possible to learn how to translate gradually and with experience (including revisiting and re-learning some aspects of the languages). We propose a similar strategy by introducing the con- cept of Scheduled Multi-Task Learning (Section 4) in which we propose to interleave the different tasks. In this paper, we propose to learn the structure of language (through syntactic parsing and part-of- speech tagging) with a multi-task learning strategy with the intentions of improving the performance of tasks like machine translation that use that structure and make generalizations. We achieve …

Content truncated to 500 words