Improving Neural Machine Translation with Soft Template Prediction

Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 5979–5989 July 5 - 10, 2020. c⃝2020 Association for Computational Linguistics 5979 Improving Neural Machine Translation with Soft Template Prediction Jian Yang1 ∗, Shuming Ma2, Dongdong Zhang2, Zhoujun Li1 †, Ming Zhou2 1State Key Lab of Software Development Environment, Beihang University 2Microsoft Research Asia {jiaya, lizj}@buaa.edu.cn; {shumma, dozhang, mingzhou}@microsoft.com; Abstract Although neural machine translation (NMT) has achieved significant progress in recent years, most previous NMT models only de- pend on the source text to generate translation. Inspired by the success of template-based and syntax-based approaches in other fields, we propose to use extracted templates from tree structures as soft target templates to guide the translation procedure. In order to learn the syntactic structure of the target sentences, we adopt the constituency-based parse tree to gen- erate candidate templates. We incorporate the template information into the encoder-decoder framework to jointly utilize the templates and source text. Experiments show that our model significantly outperforms the baseline models on four benchmarks and demonstrate the effec- tiveness of soft target templates. 1 Introduction Recently, neural machine translation (NMT) (Wu et al., 2016; Gehring et al., 2017; Vaswani et al., 2017; Chen et al., 2018) has achieved significant progress. Some advanced models (Chatterjee et al., 2016; Niehues et al., 2016; Junczys-Dowmunt and Grundkiewicz, 2017; Geng et al., 2018; Zhou et al., 2019a) predict the ultimate translation by multi-pass generation conditioned on the previous text such as CMLMs (Ghazvininejad et al., 2019), ABD-NMT (Zhang et al., 2018), SynST (Akoury et al., 2019), and Deliberation Network (Xia et al., 2017). Inspired by these works and the successful ap- plication of templates for other intriguing tasks, in- cluding semantic parsing (Dong and Lapata, 2018), summarization (Cao et al., 2018; Wang et al., 2019a), question answering (Duan et al., 2017; Asia.∗Contribution during internship at Microsoft Research †Corresponding author. I like playing basketball 我 喜欢 打 篮球 Source S like VP Template Target Figure 1: An example of template guided translation results. S denotes subject and VP denotes verb phrase. Pandey et al., 2018), and other text generation tasks (Wiseman et al., 2018; Guu et al., 2018), we assume the candidate templates of the target sentences can guide the sentence translation process. We denote these templates extracted from the constituency- based parse tree as soft templates, which consist of tags and target words. The templates are soft because no explicit paradigms are inaugurated to build new translation from them, and the target tokens could be modified. In order to effectively use the templates, we in- troduce soft template-based neural machine trans- lation (ST-NMT), which can use source text and soft templates to predict the final translation. Our approach can be split into two phases. In the first phase, a standard Transformer model is trained to predict soft target templates by using source text and templates extracted from the constituency- based parse tree. Secondly, we use two encoders, including a soft target template encoder and a source language …

Content truncated to 500 words