Robust Multilingual Part-of-Speech Tagging via Adversarial Training

Proceedings of NAACL-HLT 2018, pages 976–986 New Orleans, Louisiana, June 1 - 6, 2018. c⃝2018 Association for Computational Linguistics Robust Multilingual Part-of-Speech Tagging via Adversarial Training Michihiro Yasunaga Jungo Kasai Dragomir Radev Department of Computer Science, Yale University {michihiro.yasunaga,jungo.kasai,dragomir.radev}@yale.edu Abstract Adversarial training (AT)1 is a powerful reg- ularization method for neural networks, aim- ing to achieve robustness to input perturba- tions. Yet, the specific effects of the robust- ness obtained from AT are still unclear in the context of natural language processing. In this paper, we propose and analyze a neural POS tagging model that exploits AT. In our ex- periments on the Penn Treebank WSJ corpus and the Universal Dependencies (UD) dataset (27 languages), we find that AT not only im- proves the overall tagging accuracy, but also 1) prevents over-fitting well in low resource languages and 2) boosts tagging accuracy for rare/unseen words. We also demonstrate that 3) the improved tagging performance by AT contributes to the downstream task of depen- dency parsing, and that 4) AT helps the model to learn cleaner word representations. 5) The proposed AT model is generally effective in different sequence labeling tasks. These posi- tive results motivate further use of AT for nat- ural language tasks. 1 Introduction Recently, neural network-based approaches have become popular in many natural language pro- cessing (NLP) tasks including tagging, parsing, and translation (Chen and Manning, 2014; Bah- danau et al., 2015; Ma and Hovy, 2016). How- ever, it has been shown that neural networks tend to be locally unstable and even tiny perturba- tions to the original inputs can mislead the models (Szegedy et al., 2014). Such maliciously perturbed inputs are called adversarial examples. Adversar- ial training (Goodfellow et al., 2015) aims to im- prove the robustness of a model to input perturba- tions by training on both unmodified examples and adversarial examples. Previous work (Goodfellow 1We distinguish AT from Generative Adversarial Net- works (GANs). Figure 1: Illustration of our architecture for adversar- ial POS tagging. Given a sentence, we input the nor- malized word embeddings (w1, w2, w3) and character embeddings (showing c1, c2, c3 for w1). Each word is represented by concatenating its word embedding and its character-level BiLSTM output. They are fed into the main BiLSTM-CRF network for POS tagging. In adversarial training, we compute and add the worst- case perturbation η to all the input embeddings for reg- ularization. et al., 2015; Shaham et al., 2015) on image recog- nition has demonstrated the enhanced robustness of their models to unseen images via adversarial training and has provided theoretical explanations of the regularization effects. Despite its potential as a powerful regularizer, adversarial training (AT) has yet to be explored ex- tensively in natural language tasks. Recently, Miy- ato et al. (2017) applied AT on text classification, achieving state-of-the-art accuracy. Yet, the spe- cific effects of the robustness obtained from AT are still unclear in the context of NLP. For example, research studies have yet to answer questions such as 1) how can we …

Content truncated to 500 words