A Neural Network Model for Part-Of-Speech Tagging of Social Media Texts

A Neural Network Model for Part-Of-Speech Tagging of Social Media Texts Sara Meftah∗, Nasredine Semmar∗, Fatiha Sadat+ ∗CEA, LIST, Vision and Content Engineering Laboratory F-91191, Gif-sur-Yvette, France {sara.meftah, nasredine.semmar}@cea.fr +Universit´e du Qu´ebec `a Montr´eal, UQ `AM 201 Pr´esident Kennedy Avenue H2X 3Y7, Montr´eal, Canada sadat.fatiha@uqam.ca Abstract In this paper, we propose a neural network model for Part-Of-Speech (POS) tagging of User-Generated Content (UGC) such as Twitter, Facebook and Web forums. The proposed model is end-to-end and uses both character and word level representations. Character level representations are learned during the training of the model through a Convolutional Neural Network (CNN). For word level representations, we combine several pre-trainned embeddings (Word2Vec, FastText and GloVe). To deal with the issue of the poor availability of annotated social media data, we have implemented a Transfer Learning (TL) approach. We demonstrate the validity and genericity of our model on a POS tagging task by conducting our experiments on five social media languages (English, German, French, Italian and Spanish). Keywords: Part-of-Speech Tagging, Social Media Texts, Low-Resources Languages, Neural Networks, Transfer Learning 1. Introduction Recent approaches based on end-to-end Deep Neural Net- works (DNNs) have shown promising results for Natu- ral Language Processing (NLP). Most of proposed neural models for sequence labeling (including POS taggers) use Recurrent Neural Networks (RNNs) and its variants (Long Short-Term Memory networks - LSTMs and Gated Recur- rent Units - GRUs), and Convolutional Neural Networks (CNNs) for character-level representations. Indeed, previ- ous studies (Jozefowicz et al., 2016) have shown that CNNs represent an effective approach to extract morphological in- formation (root, prefix, suffix, etc.) from words and encode it into neural representations, especially for morphological rich texts (Chiu and Nichols, 2015; Ma and Hovy, 2016). The actual performance of POS taggers trained from tree- banks in the newswire domain, such as the Wall Street Journal (WSJ) corpus of the Penn TreeBank (PTB) (Mar- cus et al., 1993) and evaluated on in-domain data is close to human level, thanks to deep learning techniques trained on huge annotated datasets (97.64% accuracy by (Choi, 2016)). Contrariwise, approaching human-level accuracy on more complex domains such as User Generated Con- tent (UGC) on social media is still a hard problem. Espe- cially conversational texts (Twitter, Web blogs, SMS texts, etc.). This is due to the conversational nature of the text, the lack of conventional orthography, the noise, linguistic errors, spelling inconsistencies, informal abbreviations and the idiosyncratic style. Also, Twitter poses an additional issue by imposing 280 characters limit for each tweet. The application of models trained on well-structured cor- pora such as WSJ fails to work effectively on noisy text. As illustrated in (Gimpel et al., 2011), the accuracy of the Stanford POS tagger (Toutanova et al., 2003) trained on WSJ falls from 97% on standard English to 85% accuracy on tweets. The main reason for this drop in accuracy is that tweets contain lot of Out-Of-Vocabulary (OOV) words compared to standard text. In addition, NLP’s DNNs mod- els often require to be trained on huge …

Content truncated to 500 words