How Robust Are Character-Based Word Embeddings in Tagging and MT Against Wrod Scramlbing or Randdm Nouse?

How Robust Are Character-Based Word Embeddings in Tagging and MT Against Wrod Scramlbing or Randdm Nouse? Georg Heigold georg.heigold@dfki.de Stalin Varanasi Stalin.Varanasi@dfki.de G¨unter Neumann Guenter.Neumann@dfki.de Josef van Genabith Josef.Van Genabith@dfki.de DFKI, Saarbr¨ucken, Germany Abstract This paper investigates the robustness of NLP against perturbed word forms. While neural approaches can achieve (almost) human-like accuracy for certain tasks and conditions, they often are sensitive to small changes in the input such as non-canonical input (e.g., typos). Yet both stability and robustness are desired properties in applications involving user-generated content, and all the more so as humans easily cope with such noisy or adversary conditions. In this paper, we study the impact of noisy input. We consider different noise distributions (different density and different types) and mismatched noise distributions for training and test- ing. Moreover, we empirically evaluate the robustness of different models (convolutional neu- ral networks, recurrent neural networks, non-neural models), different basic units (characters, byte pair encoding units, and words), and different NLP tasks (morphological tagging, machine translation). Our experiments confirm that (i) noisy input substantially degrades the output of models trained on clean data, that (ii) training on noisy data can help models achieve perfor- mance on noisy data similar to that of models trained on clean data tested on clean data, that (iii) models trained noisy data can achieve good results on noisy data almost without performance loss on clean data, that (iv) error type mismatches between training and test data can have a greater impact than error density mismatches, that (v) character based approaches are almost always better than byte pair encoding (BPE) approaches with noisy data, that (vi) the choice of neural models (recurrent, convolutional) is not significant, and that (vii) for morphological tagging, under the same data conditions, the neural models outperform a conditional random field (CRF) based model. 1 Introduction In this paper, we study the effect of non-normalized text on natural language processing (NLP). Non-normalized text includes non-canonical word forms, noisy word forms, and word forms with ”small” perturbations, such as informal spellings, typos, scrambled words. Compared to normalized text, the variability of non-normalized text is much greater and aggravates the problem of data sparsity. Non-normalized text dominates in many real world applications. Similar to humans, ide- ally NLP should perform reliably and robustly also under suboptimal or even adversarial condi- tions, without a significant degradation in performance. Web-based content and social media are a rich source for noisy and informal text. Noise can also be introduced in a downstream NLP Proceedings of AMTA 2018, vol. 1: MT Research Track Boston, March 17 - 21, 2018 | Page 68 application where errors are propagated from one module to the next. For example, speech translation where the machine translation (MT) module needs to be robust against errors in- troduced by the automatic speech recognition (ASR) module. Moreover, NLP should not be vulnerable to adversarial input examples. While all these examples do not pose a real challenge to an experienced human reader, even ”small” perturbations from …

Content truncated to 500 words