Learning to Segment Inputs for NMT Favors Character-Level Processing

Learning to Segment Inputs for NMT Favors Character-Level Processing Julia Kreutzer∗, Artem Sokolov⋄,∗ ∗Computational Linguistics, Heidelberg University, Germany ⋄Amazon Research, Germany kreutzer@cl.uni-heidelberg.de, artemsok@amazon.com Abstract Most modern neural machine translation (NMT) systems rely on presegmented inputs. Segmentation granularity im- portantly determines the input and output sequence lengths, hence the modeling depth, and source and target vocabu- laries, which in turn determine model size, computational costs of softmax normalization, and handling of out-of- vocabulary words. However, the current practice is to use static, heuristic-based segmentations that are fixed before NMT training. This begs the question whether the chosen segmentation is optimal for the translation task. To over- come suboptimal segmentation choices, we present an algo- rithm for dynamic segmentation, that is trainable end-to-end and driven by the NMT objective. In an evaluation on four translation tasks we found that, given the freedom to navi- gate between different segmentation levels, the model prefers to operate on (almost) character level, providing support for purely character-level NMT models from a novel angle. 1. Introduction Segmentation of input sequences is an essential preprocess- ing step for neural machine translation (NMT) and has been found to have a high positive impact on translation quality in recent WMT shared task evaluations [1, 2]. This success can be explained statistically, since shorter segments are benefi- cial for reducing sparsity: They lower the type-to-token ratio, decrease the number of out-of-vocabulary (OOV) tokens and singletons, which improves the coverage of unseen inputs. Two subword segmentation methods are presently the state-of-the-art in NMT: the byte-pair encoding (BPE), that starts with a dictionary of single characters and iteratively creates a new entry from the two currently most frequent en- tries [3, 4], and a similar wordpiece (WP) model [5]. While being empirically more successful than word- based NMT, both BPE and WP are preprocessing heuristics, they do not account for the translation task or the language pairs at hand (unless applied to both sides jointly), and re- quire additional preprocessing for languages that lack ex- plicit word separation in writing. Being used in a pipeline fashion, they make it impossible for an NMT system to re- segment an unfavorably presplit input and require consistent Work was done while the first author was interning at Amazon, Berlin. application of the same segmentation model during testing, which adds an integration overhead and contributes to the ‘pipeline jungles’ in production environments [6]. On the other extreme from word-based NMT models lie purely character models. Their advantages are smaller vo- cabularies, thus smaller embedding and output layers, allow- ing for more learning iterations within a training time budget to improve generalization [7], and no preprocessing require- ments. At the same time, longer input sequences aggravate known optimization problems with very large depths of time- unrolled RNNs [8] and may require additional memory for tracking gradients along the unrolling steps. In this work1, we pose the following question: what would the input segmentations look like if the NMT model could decide on them dynamically? Instead of heuristically committing …

Content truncated to 500 words