Segmenting Hashtags using Automatically Created Training Data

Segmenting Hashtags using Automatically Created Training Data Arda C¸ elebi and Arzucan ¨Ozg¨ur Department of Computer Engineering Bo˘gazic¸i University Bebek, 34342 Istanbul, Turkey { arda.celebi, arzucan.ozgur } @boun.edu.tr Abstract Hashtags, which are commonly composed of multiple words, are increasingly used to convey the actual messages in tweets. Under- standing what tweets are saying is getting more dependent on understanding hashtags. Therefore, identifying the individual words that constitute a hashtag is an important, yet a challenging task due to the abrupt nature of the language used in tweets. In this study, we introduce a feature-rich approach based on using supervised machine learning methods to segment hashtags. Our approach is unsupervised in the sense that instead of using manually segmented hashtags for training the machine learning classifiers, we automatically create our training data by using tweets as well as by automatically extracting hashtag segmentations from a large corpus. We achieve promising results with such automatically created noisy training data. Keywords: Social Media Processing, Hashtags, Segmentation 1. Introduction Word segmentation is a well-known problem in the field of Natural Language Processing (NLP) with a number of ap- plication fields including tokenization in languages such as Chinese and Japanese, speech recognition, and URL seg- mentation. All these well-studied cases aside, the introduction of hash- tags on social media platforms including Twitter and In- stagram creates a new challenge. Originally, hashtags are introduced as labels to assign contexts to tweets. They start with the ‘#’sign, followed by one or more words concate- nated one after another. While some of them consist of single words, people increasingly create hashtags out of complex phrases and even whole sentences. Since hashtags indicate the contexts of tweets and sometimes even act as the actual messages conveyed in the tweets, breaking them into their constituent words is essential for understanding tweets. In this study, we develop a feature-rich supervised ma- chine learning based approach for hashtag segmentation. Unlike the traditional supervised setting, where training is performed using manually annotated training data, we uti- lize the large amount of unlabeled data available in the so- cial media and use two approaches to automatically cre- ate our training data. In this respect, our approach can be considered as unsupervised. First, we use normalized tweets to create synthetic hashtag segmentations. Second, we employ a method which automatically extracts hash- tag segmentations from a large set of tweets. As a side- contribution of this paper, we also create and share1 two data sets each one consisting of 1000 manually segmented hashtags. 2. Related Work Word segmentation methods can be as simple as using a vo- cabulary to find words. One of the best known vocabulary- based methods is maximum matching (Wong and Chan, 1http://tabilab.cmpe.boun.edu.tr/projects/hashtag segmentation 1996) and its variations such as the greedy algorithm used in (Dale et al., 2010). Many state-of-the-art systems em- ploy statistical approaches. In general, they are more suc- cessful at handling unknown words and picking the best possible alternative in case of ambiguity. Such methods treat the problem as …

Content truncated to 500 words