Prior Knowledge Integration for Neural Machine Translation using Posterior Regularization

Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics, pages 1514–1523 Vancouver, Canada, July 30 - August 4, 2017. c⃝2017 Association for Computational Linguistics https://doi.org/10.18653/v1/P17-1139 Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics, pages 1514–1523 Vancouver, Canada, July 30 - August 4, 2017. c⃝2017 Association for Computational Linguistics https://doi.org/10.18653/v1/P17-1139 Prior Knowledge Integration for Neural Machine Translation using Posterior Regularization Jiacheng Zhang†, Yang Liu†‡∗, Huanbo Luan†, Jingfang Xu# and Maosong Sun†‡ †State Key Laboratory of Intelligent Technology and Systems Tsinghua National Laboratory for Information Science and Technology Department of Computer Science and Technology, Tsinghua University, Beijing, China ‡Jiangsu Collaborative Innovation Center for Language Competence, Jiangsu, China #Sogou Inc., Beijing, China Abstract Although neural machine translation has made significant progress recently, how to integrate multiple overlapping, arbitrary prior knowledge sources remains a chal- lenge. In this work, we propose to use pos- terior regularization to provide a general framework for integrating prior knowl- edge into neural machine translation. We represent prior knowledge sources as fea- tures in a log-linear model, which guides the learning process of the neural trans- lation model. Experiments on Chinese- English translation show that our approach leads to significant improvements. 1 Introduction The past several years have witnessed the rapid de- velopment of neural machine translation (NMT) (Sutskever et al., 2014; Bahdanau et al., 2015), which aims to model the translation process using neural networks in an end-to-end manner. With the capability of capturing long-distance depen- dencies due to the gating (Hochreiter and Schmid- huber, 1997; Cho et al., 2014) and attention (Bah- danau et al., 2015) mechanisms, NMT has shown remarkable superiority over conventional statisti- cal machine translation (SMT) across a variety of natural languages (Junczys-Dowmunt et al., 2016). Despite the apparent success, NMT still suffers from one significant drawback: it is difficult to in- tegrate prior knowledge into neural networks. On one hand, neural networks use continuous real- valued vectors to represent all language structures involved in the translation process. While these vector representations prove to be capable of cap- turing translation regularities implicitly (Sutskever ∗Corresponding author: Yang Liu. et al., 2014), it is hard to interpret each hidden state in neural networks from a linguistic perspective. On the other hand, prior knowledge in machine translation is usually represented in discrete sym- bolic forms such as dictionaries and rules (Niren- burg, 1989) that explicitly encode translation regu- larities. It is difficult to transform prior knowledge represented in discrete forms to continuous repre- sentations required by neural networks. Therefore, a number of authors have endeav- ored to integrate prior knowledge into NMT in re- cent years, either by modifying model architec- tures (Tu et al., 2016; Cohn et al., 2016; Tang et al., 2016; Feng et al., 2016) or by modifying training objectives (Cohn et al., 2016; Feng et al., 2016; Cheng et al., 2016). For example, to address the over-translation and under-translation prob- lems widely observed in NMT, Tu et al. (2016) directly extend standard NMT to model the …

Content truncated to 500 words