Toward Robust Neural Machine Translation for Noisy Input Sequences

Toward Robust Neural Machine Translation for Noisy Input Sequences Matthias Sperber, Jan Niehues, Alex Waibel Interactive Systems Labs Karlsruhe Institute of Technology, Germany matthias.sperber@kit.edu Abstract Translating noisy inputs, such as the output of a speech rec- ognizer, is a difficult but important challenge for neural ma- chine translation. One way to increase robustness of neu- ral models is by introducing artificial noise to the training data. In this paper, we experiment with appropriate forms of such noise, exploring a middle ground between general- purpose regularizers and highly task-specific forms of noise induction. We show that with a simple generative noise model, moderate gains can be achieved in translating erro- neous speech transcripts, provided that type and amount of noise are properly calibrated. The optimal amount of noise at training time is much smaller than the amount of noise in our test data, indicating limitations due to trainability issues. We note that unlike our baseline model, models trained on noisy data are able to generate outputs of proper length even for noisy inputs, while gradually reducing output length for higher amount of noise, as might also be expected from a hu- man translator. We discuss these findings in details and give suggestions for future work. 1. Introduction Many natural language processing tasks require applying se- quence models on corrupted or noisy input sequences. A typical example is machine translation of erroneous outputs from an automatic speech recognizer (ASR). Ideally, we would like the translation process to ignore or even correct the corrupted inputs. Translation models are usually trained on wellformed parallel sentences that do not exhibit such noise. This results in a harmful mismatch between training and test data, and further aggravates the difficulty of hav- ing to transform malformed inputs in the first place. The now prevalent neural sequence-to-sequence models [1, 2, 3] have been identified to be especially sensitive to noisy data [4, 5, 6], and more specifically to corrupted inputs due to er- roneous ASR [7]. Robustness at test-time may be improved by inducing suitable forms of noise during the training process. The spec- trum of suitable approaches ranges from general-purpose regularizers1 such as dropout [9] to task-specific approaches 1In this paper, we use the notions of good generalization (avoiding over- fitting, e.g. via regularization) and robustness (stability w.r.t. noisy data) that alter the training data to resemble the corrupted inputs at test-time. Task-specific approaches can make stronger assumptions about the data distribution and are potentially more effective or provide additive gains when combined with general-purpose methods. As a disadvantage, they are also more complex and may require task-specific knowledge or resources. Another tradeoff to consider concerns trainabil- ity. Neural sequence-to-sequence models are known to suf- fer from explaining-away effects, where models may learn to generate outputs by relying on the target-side context while ignoring the source-side context [10, 11], especially when the source side provides only a weak or noisy signal. As a result, careful calibration of type and amount of induced noise may be necessary. Prior …

Content truncated to 500 words