Classification of Micro-Texts Using Sub-Word Embeddings

Proceedings of Recent Advances in Natural Language Processing, pages 526–533, Varna, Bulgaria, Sep 2–4, 2019. https://doi.org/10.26615/978-954-452-056-4_062 526 Classification of Micro-Texts Using Sub-Word Embeddings Mihir Joshi Faculty of Computer Science Dalhousie University Halifax, Nova Scotia, Canada mihir@dal.ca Nur Zincir-Heywood Faculty of Computer Science Dalhousie University Halifax, Nova Scotia, Canada zincir@cs.dal.ca Abstract Extracting features and writing styles from short text messages is always a challenge. Short messages, like tweets, do not have enough data to perform statistical author- ship attribution. Besides, the vocabulary used in these texts is sometimes impro- vised or misspelled. Therefore, in this paper, we propose combining four fea- ture extraction techniques namely charac- ter n-grams, word n-grams, Flexible Pat- terns and a new sub-word embedding us- ing the skip-gram model. Our system uses a Multi-Layer Perceptron to utilize these features from tweets to analyze short text messages. This proposed system achieves 85% accuracy, which is a considerable im- provement over previous systems. 1 Introduction One of the most challenging problems in text anal- ysis is identifying the author of a micro-text, i.e. short text messages. Most of the data created on social media applications whether a Tweet, Face- book comment or text on a messaging application is a micro text. A micro or short text message could be a tweet or a comment which is around 140 characters or less. In general, text analysis and natural language processing approaches em- ploy statistical feature extraction techniques such as term frequency, inverse document frequency and a bag of words. Due to the feature extraction process being statistical in nature, all these tech- niques require a certain amount of data to use them effectively for determining patterns or perform au- thorship attribution. Thus, having short text mes- sages such as tweets which is around 140 charac- ters or less makes it difficult to identify the pat- terns on a given text and make predictions about the author. Shrestha et al. (2017) used a convolutional neu- ral network (CNN) architecture using character embeddings instead of word embeddings for short texts. With this approach they showed less than five percent improvement on previous researches (Qian et al., 2015; Schwartz et al., 2013) in this area. In this paper, we start by implementing a simpler approach; combining the two consecutive records from the same author to train our model and then apply the feature known as Flexible pat- terns (Schwartz et al., 2013) after modifying the existing method and finally introducing our new feature based on the word vector approach (Bo- janowski et al., 2017). The paper is organized in the following sec- tions. Section 2 represents related works. In section 3 we define our model architecture fol- lowed by the dataset and the feature extraction techniques. Section 4 shows the results of our ap- proach compared with previous works. Lastly, in section 5, we discuss conclusion and future work. 2 Related Works In short-text analysis, one of the earlier works by Layton et al. (2010) aims to identify the author based on the …

Content truncated to 500 words