Improved Lexically Constrained Decoding for Translation and Monolingual Rewriting

Proceedings of NAACL-HLT 2019, pages 839–850 Minneapolis, Minnesota, June 2 - June 7, 2019. c⃝2019 Association for Computational Linguistics 839 Improved Lexically Constrained Decoding for Translation and Monolingual Rewriting J. Edward Hu Huda Khayrallah Ryan Culkin Patrick Xia Tongfei Chen Matt Post Benjamin Van Durme Johns Hopkins University {edward.hu,huda,rculkin,paxia,tongfei}@jhu.edu; {post,vandurme}@cs.jhu.edu Abstract Lexically-constrained sequence decoding al- lows for explicit positive or negative phrase- based constraints to be placed on target out- put strings in generation tasks such as machine translation or monolingual text rewriting. We describe vectorized dynamic beam allocation, which extends work in lexically-constrained decoding to work with batching, leading to a five-fold improvement in throughput when working with positive constraints. Faster de- coding enables faster exploration of constraint strategies: we illustrate this via data augmenta- tion experiments with a monolingual rewriter applied to the tasks of natural language infer- ence, question answering and machine transla- tion, showing improvements in all three. 1 Introduction For many natural language generation tasks, we often know word(s) that should (or should not) be in the output sentence. Examples include ter- minology databases in Machine Translation (MT) (Hokamp and Liu, 2017), names (and generic re- sponses) in dialogue generation (Li et al., 2016; Gu et al., 2016), objects in image captioning (An- derson et al., 2017), and facts in abstractive sum- marization (See et al., 2017). One approach to en- force hard lexical constraints in the output is to modify the inference procedure to enforce their presence directly (Hokamp and Liu, 2017). These constraints could be either positive (a word must appear in the output) or negative (a word must be avoided). While negative constraints could be easily enforced by preventing hypotheses with prohibited tokens from entering the beam, plac- ing positive constraints in natural and meaningful ways is less straightforward. We improve upon previous work by vectorizing the dynamic beam allocation (DBA) algorithm from Post and Vi- lar (2018) and by incorporating multi-state tries, which track a subset of nodes at each decoding timestep. These improvements lead to a five-fold speedup in decoding with positive constraints and in some cases better constraint placements (with respect to BLEU). Post and Vilar (2018) motivated the util- ity of lexically-constrained decoding in MT for scenarios such as interactive translation and do- main adaptation. Translation applications han- dling large amounts of data will clearly benefit from improvements in speed: the same is true for large-scale data augmentation via rewriting. In this case, a practitioner will ideally explore various task-specific rewriting strategies that may lead to improvements as observed during devel- opment, and then incorporate the best strategy into a test-final model. Recently, sentential para- phrasing gained the ability to enforce lexical con- straints (Hu et al., 2019), but constrained decoding was still too inefficient to be practical (Hokamp and Liu, 2017) at a large scale. Even with the approach described by Post and Vilar, exploring the space of possible rewriting strategies on a task- specific basis may be overly time consuming: our performance improvements to their algorithm …

Content truncated to 500 words