Tagging The Web: Building A Robust Web Tagger with Neural Network

Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics, pages 144–154, Baltimore, Maryland, USA, June 23-25 2014. c⃝2014 Association for Computational Linguistics Tagging The Web: Building A Robust Web Tagger with Neural Network Ji Ma†, Yue Zhang‡ and Jingbo Zhu† †Northeastern University, China ‡Singapore University of Technology and Design majineu@gmail.com yue zhang@sutd.edu.sg zhujingbo@mail.neu.edu.cn Abstract In this paper, we address the problem of web-domain POS tagging using a two- phase approach. The first phase learns rep- resentations that capture regularities un- derlying web text. The representation is integrated as features into a neural network that serves as a scorer for an easy-first POS tagger. Parameters of the neural network are trained using guided learning in the second phase. Experiment on the SANCL 2012 shared task show that our approach achieves 93.15% average tagging accu- racy, which is the best accuracy reported so far on this data set, higher than those given by ensembled syntactic parsers. 1 Introduction Analysing and extracting useful information from the web has become an increasingly important re- search direction for the NLP community, where many tasks require part-of-speech (POS) tag- ging as a fundamental preprocessing step. How- ever, state-of-the-art POS taggers in the literature (Collins, 2002; Shen et al., 2007) are mainly opti- mized on the the Penn Treebank (PTB), and when shifted to web data, tagging accuracies drop sig- nificantly (Petrov and McDonald, 2012). The problem we face here can be considered as a special case of domain adaptation, where we have access to labelled data on the source domain (PTB) and unlabelled data on the target domain (web data). Exploiting useful information from the web data can be the key to improving web domain tagging. Towards this end, we adopt the idea of learning representations which has been demonstrated useful in capturing hidden regular- ities underlying the raw input data (web text, in our case). Our approach consists of two phrases. In the pre-training phase, we learn an encoder that con- verts the web text into an intermediate represen- tation, which acts as useful features for prediction tasks. We integrate the learned encoder with a set of well-established features for POS tagging (Rat- naparkhi, 1996; Collins, 2002) in a single neural network, which is applied as a scorer to an easy- first POS tagger. We choose the easy-first tagging approach since it has been demonstrated to give higher accuracies than the standard left-to-right POS tagger (Shen et al., 2007; Ma et al., 2013). In the fine-tuning phase, the parameters of the network are optimized on a set of labelled train- ing data using guided learning. The learned model preserves the property of preferring to tag easy words first. To our knowledge, we are the first to investigate guided learning for neural networks. The idea of learning representations from un- labelled data and then fine-tuning a model with such representations according to some supervised criterion has been studied before (Turian et al., 2010; Collobert et al., 2011; Glorot et al., 2011). While most …

Content truncated to 500 words