Processing and Normalizing Hashtags

Proceedings of Recent Advances in Natural Language Processing, pages 104–109, Hissar, Bulgaria, Sep 7–9 2015. Processing and Normalizing Hashtags Thierry Declerck Dept. of Computational Linguistics, Saarland University, Saarbrücken, Germany declerck@dfki.de Piroska Lendvai Dept. of Computational Linguistics, Saarland University, Saarbrücken, Germany piroska.r@gmail.com Abstract We present ongoing work in linguistic pro- cessing of hashtags in Twitter text, with the goal of supplying normalized hashtag content to be used in more complex natural language processing (NLP) tasks. Hashtags represent collectively shared topic designators with considerable surface variation that can ham- per semantic interpretation. Our normaliza- tion scripts allow for the lexical consolidation and segmentation of hashtags, potentially leading to improved semantic classification. 1 Introduction The relevance of hashtags used in social media text, and more specifically in Twitter messages, has been recognized after some studies focused on the semantics that can be derived by such text constructs. For example, Laniado & Mika (2010) discussed whether hashtags behave as identifiers for Semantic Web applications. But the authors do not raise the issue of processing the hashtags in order to harmonize them, which would be necessary for gaining information on the specific semantics carried by hashtags. Our observations are based on a large Twitter corpus dedicated to riots in the UK in the sum- mer of 20111.Variants for hashtags that refer to the same topic abound, e.g. “#LondonRiots”, “#londonriots”, “#RiotsInLondon”, “londonriot”, 1 This corpus was built on behalf of the newspaper „The Guardian“, and its first objective was to gather data for tra- cking the emergence of rumours in social media. See http://www.theguardian.com/news/datablog/2011/dec/08/tw itter-riots-interactive . An example usage of this corpus with NLP approaches for argumentation research is given in (Llewellyn et al., 2014). “#LONDONRIOTS”, and so on. We hypothesize that consolidating variants to a preferred hashtag form would benefit further tasks that draw on semantic similarity, such as the recently orga- nized Semantic Textual Similarity Shared Task on Twitter data2. We have implemented a set of scripts in order to normalize the surface forms of hashtags. This includes case normalization, lemmatization and syntactic segmentation. We first describe related work, then our approach, and finally display some of our current results. 2 Related Work and Task Specification (Pöschko, 2011) focuses on the detection of simi- lar hashtags on the basis of their co-occurrences in a tweet. While the detection of co-occurrences is also present in our pipeline, we are additional- ly interested in detecting variants in order to re- duce the amount of topics that hashtags desig- nate. The expectation is that hashtag variants would all together represent only one topic. (Antenucci et al., 2011) discuss an algorithm to learn the relationships between the literal con- tent of a tweet and the types of hashtags that de- scribe that content, which is one of our goals as well. Contrary to us, (Antenucci et al., 2011) do not suggest the harmonization (or reduction to a preferred form) of hashtags, but use similarity measurements between hashtags and words, while we implement patterns for explicitly relat- ing variants …

Content truncated to 500 words