An Exploration of Placeholding in Neural Machine Translation

An Exploration of Placeholding in Neural Machine Translation Matt Post†⋄ Shuoyang Ding† Marianna J. Martindale‡ Winston Wu† † Center for Language and Speech Processing, Johns Hopkins University ‡ iSchool, University of Maryland College Park ⋄ Human Language Technology Center of Excellence, Johns Hopkins University post@cs.jhu.edu, {dings, wswu}@jhu.edu mmartind@umiacs.umd.edu Abstract Phrase-based machine translation provides the system developer with controls that enable fine-grained control over machine translation output. One approach to provide similar control in neural machine transla- tion is placeholding (herein called mask- ing), which replaces input tokens with masks which are replaced with the origi- nal input text in post-processing. But is this a good idea? We undertake an explo- ration of masking in French–English and Japanese–English using Transformer archi- tectures. We attempt to quantify whether (and where) masking is necessary with anal- ysis of a baseline system, and then explore numerous parameterization of masking, in- cluding post-processing techniques for re- placing the masks. Our analysis shows this to be a thorny matter; masks solve some problems but are not perfectly translated themselves. 1 Introduction Neural machine translation generally produces higher quality output than phrase-based machine translation, especially in high-resource training set- tings and on in-domain data. However, this im- provement has come at the expense of a certain loss of control over how words get translated, since there is no longer a direct link between source words, their translation options, and the ordered decoder output. While nearly everyone has con- sidered this trade to be worthwhile, there lingers c⃝ 2019 The authors. This article is licensed under a Creative Commons 4.0 licence, no derivative works, attribution, CC- BY-ND. src En 2017 Bernard Arnault a gagn´e... mask En NUM NAME NAME a gagn´e... out In NUM NAME NAME a gagn´e... align In NUM1 NAME1 NAME2 a gagn´e... detok In 2017 Bernard Arnault won... Figure 1: A translation pipeline with masking (placeholding). The indexes denote a permutation of each mask type, and may or may not be an explicit part of the tag. a concern about the stability and dependency of NMT performance. Input words are not all equally important, and there are many settings where one would be willing to sacrifice translation quality for a translation guarantee that certain input tokens be translated with perfect recall. Common exam- ples include prices on a product page, names and places in a news article, or contact and location information, and other data types, such as URLs. One attempt to providing these guarantees is the use of placeholders (or masks, the term we will use in this paper), where input tokens in a category are replaced by a masked label token (Figure 1). These are then passed through to the output and replaced with the correct translation in post-processing. This ostensibly guarantees that the input term (or its preferred translation) will correctly appear in the output, while at the same time restoring a capability that was easily handled in the old phrase-based paradigm. At the same time, doing so reflects a lack of …

Content truncated to 500 words