Exploration of Noise Strategies in Semi-supervised Named Entity Classification

Proceedings of the Eighth Joint Conference on Lexical and Computational Semantics (*SEM), pages 186–191 Minneapolis, June 6–7, 2019. c⃝2019 Association for Computational Linguistics 186 Exploration of Noise Strategies in Semi-supervised Named Entity Classification Pooja Lakshmi Narayan University of Arizona poojal@email.arizona.edu Ajay Nagesh∗ DiDi AI Labs ajaynagesh@didiglobal.com Mihai Surdeanu University of Arizona msurdeanu@email.arizona.edu Abstract Noise is inherent in real world datasets and modeling noise is critical during training as it is effective in regularization. Recently, novel semi-supervised deep learning tech- niques have demonstrated tremendous poten- tial when learning with very limited labeled training data in image processing tasks. critical aspect of these semi-supervised learn-A ing techniques is augmenting the input or the network with noise to be able to learn ro- bust models. While modeling noise is rela- tively straightforward in continuous domains such as image classification, it is not immedi- ately apparent how noise can be modeled in discrete domains such as language. Our work aims to address this gap by exploring different noise strategies for the semi-supervised named entity classification task, including statistical methods such as adding Gaussian noise to input embeddings, and linguistically-inspired ones such as dropping words and replacing words with their synonyms. We compare their performance on two benchmark datasets (OntoNotes and CoNLL) for named entity classification. Our results indicate that noise strategies that are linguistically informed per- form at least as well as statistical approaches, while being simpler and requiring minimal tuning. 1 Introduction Modeling noise is a fundamental aspect of ma- chine learning systems. The real world where these systems are deployed are certainly exposed to noisy data. Furthermore, noise is used as an effective regularizer during the training of neural networks (e.g., dropout (Srivastava et al., 2014)). Correct prediction in the presence of noisy in- put demonstrates robustness of learning systems. A simple analogy to illustrate this is, during im- age classification, the addition of limited random ∗ work done during AN’s post-doc at Univ. of Arizona Gaussian noise to an image can be barely per- ceived by our visual system and does not dras- tically change the label a human assigns to an image (Raj, 2018). With the emphasis on com- pliance and recent advances in adversarial tech- niques, modeling noise has assumed renewed im- portance (Goodfellow et al., 2014). Noise is an important factor in recent state-of- the-art semi-supervised learning systems for im- age classification (Tarvainen and Valpola, 2017; Rasmus et al., 2015; Miyato et al., 2018). In image processing modeling random noise is rel- atively straightforward as it is a continuous do- main. For instance, adding a small amount ran- dom Gaussian jitter can be considered as noisy in- put. So are other image transformations such as translation, rotation, removing color and so on. However, a discrete domain such as language is not easily amenable to noise augmentation. While one can certainly add random Gaussian noise to embeddings of words (continuous vector represen- tation such as word2vec rather than one-hot en- coding), the intuition behind such perturbation is not apparent. …

Content truncated to 500 words