Constraining the Transformer NMT Model with Heuristic Grid Beam Search

∗ Constraining the Transformer NMT Model with Heuristic Grid Beam Search Guodong Xie guodong.xie@adaptcentre.ie Andy Way andy.way@adaptcentre.ie Jinhua Du jinhua.du@adaptcentre.ie ADAPT Centre, School of Computing, Dublin City University, Dublin, Ireland Longyue Wang∗ vincentwang0229@gmail.com Tencent AI Lab Abstract Constrained decoding forces a certain set words or phrases to appear in the translation results and is very useful when adapting MT to a certain domain. In recent years, the Transformer model has outperformed other neural machine translation models to become the state-of-the- art paradigm. However, constrained decoding for domain adaptation remains an open problem under the Transformer model. In this paper, we first investigate how a constrained decoding method – Grid Beam Search (GBS) – performs in the Transformer model, and then propose a source-informed heuristic method that can fully take advantage of the alignment information from the multi-head attention mechanism in Transformer to speed up the decoding in the GBS method and guide the placement of constraints during the expansion of hypotheses in GBS. Ex- periments on English–Chinese and English–German translation domain adaptation tasks show that the proposed method significantly outperforms the basic Transformer model in terms of BLEU and METEOR score, and prunes up to 30% hypotheses to save up to 20% decoding time compared to the GBS model while maintaining comparable translation performance. 1 Introduction With recent advances in neural machine translation (NMT), The Transformer model (Vaswani et al., 2017) has outperformed other NMT architectures, like RNN (Luong et al., 2015) and CNN (Gehring et al., 2017), to become the state-of-the-art paradigm. The Transformer model mainly consists of layers of self-attention and a feed-forward network. It is capable of being fully parallelised and is faster both in training and inference. Its multi-layer and multi-head attention mechanism enable it to capture deep syntactic and semantic relations in sentences to produce better translation results. Constrained decoding is an approach that exerts some constraints to a decoding process (often a beam search process) and enforces the constraints to appear in the decoding results. For translation tasks, constraints are normally some target words or phrases which are acquired in advance through domain knowledge or other methods. Constrained decoding ensures con- straints partially or fully appear in the translation results by means of certain algorithms. The translation results of constrained decoding are often better than the normal decoding results, as those constraints are actually some external knowledge besides the source sentences. ∗This work was done while the co-author were working with us in the ADAPT Centre at Dublin City University. Proceedings of the 14th Conference of the Association for Machine Translation in the Americas October 6 - 9, 2020, Volume 1: MT Research Track Page 36 Different constrained decoding approaches have been proposed. Luong and Manning (2015), Sennrich et al. (2016a) adapted NMT systems with domain-specific data by adjust- ing their output vocabulary to better match the target domain. Besides, Wang et al. (2017) tried early attempts to improve translation consistency for NMT models with discourse-level context. However, these methods do not strictly enforce …

Content truncated to 500 words