Lexically Constrained Neural Machine Translation with Levenshtein Transformer

Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 3536–3543 July 5 - 10, 2020. c⃝2020 Association for Computational Linguistics 3536 Lexically Constrained Neural Machine Translation with Levenshtein Transformer Raymond Hendy Susanto, Shamil Chollampatt, and Liling Tan Rakuten Institute of Technology Singapore Rakuten, Inc. {first.last}@rakuten.com Abstract This paper proposes a simple and effective al- gorithm for incorporating lexical constraints in neural machine translation. Previous work ei- ther required re-training existing models with the lexical constraints or incorporating them during beam search decoding with signifi- cantly higher computational overheads. Lever- aging the flexibility and speed of a recently proposed Levenshtein Transformer model (Gu et al., 2019), our method injects terminology constraints at inference time without any im- pact on decoding speed. Our method does not require any modification to the training procedure and can be easily applied at run- time with custom dictionaries. Experiments on English-German WMT datasets show that our approach improves an unconstrained baseline and previous approaches. 1 Introduction Neural machine translation (NMT) systems can generate higher-quality translations than phrase- based MT systems, but they come at the cost of losing control over how translations are generated. Without the explicit link between the source and the target vocabulary, enforcing specific terminologi- cal translation in domain-specific settings becomes painfully difficult for NMT systems. Consider an example where we have a Chinese-English NMT system trained for the E-commerce domain, and there is no prior knowledge of the brand name “红 米” in the training data, the system would translate the input term literally as “red (红) rice (米)” in- stead of “Redmi”. In such scenarios, machine trans- lation users often maintain in-domain dictionaries to ensure that specific information is translated ac- curately and consistently. A line of previous work that tried to address this problem required re-training the NMT models with lexical constraints, either by a placeholder mecha- nism (Crego et al., 2016) or via code-mixed train- ing (Song et al., 2019; Dinu et al., 2019). However, they do not reliably guarantee the presence of the constraints at test time. Another approach focused on constrained beam search decoding (Hokamp and Liu, 2017; Post and Vilar, 2018; Hu et al., 2019). Although the latter approach has higher control over the target constraint terms, they significantly slow down the decoding. Different from the existing line of work, we in- voke lexical constraints using a non-autoregressive approach.1 To do this, we use Levenshtein Trans- former (LevT) (Gu et al., 2019), an edit-based generation model that performs deletion and inser- tion operations during inference iteratively. LevT achieves substantially higher inference speed com- pared to beam search without affecting quality. We add a constraint insertion step in LevT de- coding to seamlessly decode the target language sequence while adhering to specific lexical con- straints, achieving the same speed as standard LevT decoding. 2 Related Work Previous approaches integrated lexical constraints in NMT either via constrained training or decoding. Crego et al. (2016) replaced entities with place- holders that remained unchanged during transla- …

Content truncated to 500 words