Integrating Vectorized Lexical Constraints for Neural Machine Translation

Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics Volume 1: Long Papers, pages 7063 - 7073 May 22-27, 2022 c⃝2022 Association for Computational Linguistics Integrating Vectorized Lexical Constraints for Neural Machine Translation Shuo Wang1 Zhixing Tan1 Yang Liu1,2,3,4,5∗ 1Dept. of Comp. Sci. & Tech., Institute for AI, Tsinghua University, Beijing, China Beijing National Research Center for Information Science and Technology 2Institute for AI Industry Research, Tsinghua University, Beijing, China 3International Innovation Center of Tsinghua University, Shanghai, China 4Quan Cheng Laboratory 5Institute for Guo Qiang, Tsinghua University, Beijing, China wangshuo.thu@gmail.com {zxtan, liuyang2011}@tsinghua.edu.cn Abstract Lexically constrained neural machine transla- tion (NMT), which controls the generation of NMT models with pre-specified constraints, is important in many practical scenarios. Due to the representation gap between discrete con- straints and continuous vectors in NMT models, most existing works choose to construct syn- thetic data or modify the decoding algorithm to impose lexical constraints, treating the NMT model as a black box. In this work, we propose to open this black box by directly integrating the constraints into NMT models. Specifically, we vectorize source and target constraints into continuous keys and values, which can be uti- lized by the attention modules of NMT mod- els. The proposed integration method is based on the assumption that the correspondence be- tween keys and values in attention modules is naturally suitable for modeling constraint pairs. Experimental results show that our method consistently outperforms several representative baselines on four language pairs, demonstrating the superiority of integrating vectorized lexical constraints. 1 1 Introduction Controlling the lexical choice of the translation is important in a wide range of settings, such as interactive machine translation (Koehn, 2009), en- tity translation (Li et al., 2018), and translation in safety-critical domains (Wang et al., 2020). How- ever, different from the case of statistical machine translation (Koehn et al., 2007), it is non-trivial to directly integrate discrete lexical constraints into neural machine translation (NMT) models (Bah- danau et al., 2015; Vaswani et al., 2017), whose hidden states are all continuous vectors that are difficult for humans to understand. In accordance with this problem, one branch of studies directs its attention to designing ad- ∗Correspondence to: Yang Liu. 1For the source code, please refer to https://github. com/shuo-git/VecConstNMT. The Beatles are a great band Keys Values band 乐团 Beatles Beatles The Beatles are a great band Figure 1: An example of the integration of vectorized lexical constraints into attention proposed in this work. We omit queries for simplicity. Blue and green squares denote the continuous representation of the source sen- tence and the constraints, respectively. The provided constraints are "Beatles→Beatles" and "band→乐团". vanced decoding algorithms (Hokamp and Liu, 2017; Hasler et al., 2018; Post and Vilar, 2018) to impose hard constraints and leave NMT models unchanged. For instance, Hu et al. (2019) propose a vectorized dynamic beam allocation (VDBA) al- gorithm, which devotes part of the beam to can- didates that have met some constraints. Although this kind of method can guarantee the presence …

Content truncated to 500 words