Improving Named Entity Recognition in Tweets via Detecting Non-Standard Words

Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing, pages 929–938, Beijing, China, July 26-31, 2015. c⃝2015 Association for Computational Linguistics Improving Named Entity Recognition in Tweets via Detecting Non-Standard Words Chen Li and Yang Liu Computer Science Department, The University of Texas at Dallas Richardson, Texas 75080, USA {chenli,yangl@hlt.utdallas.edu} Abstract Most previous work of text normalization on informal text made a strong assumption that the system has already known which tokens are non-standard words (NSW) and thus need normalization. However, this is not realistic. In this paper, we propose a method for NSW detection. In addi- tion to the information based on the dic- tionary, e.g., whether a word is out-of- vocabulary (OOV), we leverage novel in- formation derived from the normalization results for OOV words to help make deci- sions. Second, this paper investigates two methods using NSW detection results for named entity recognition (NER) in social media data. One adopts a pipeline strat- egy, and the other uses a joint decoding fashion. We also create a new data set with newly added normalization annota- tion beyond the existing named entity la- bels. This is the first data set with such annotation and we release it for research purpose. Our experiment results demon- strate the effectiveness of our NSW detec- tion method and the benefit of NSW detec- tion for NER. Our proposed methods per- form better than the state-of-the-art NER system. 1 Introduction Short text messages or comments from social me- dia websites such as Facebook and Twitter have become one of the most popular communication forms in recent years. However, abbreviations, misspelled words and many other non-standard words are very common in short texts for vari- ous reasons (e.g., length limitation, need to con- vey much information, writing style). They post problems to many NLP techniques in this domain. There are many ways to improve language pro- cessing performance on the social media data. One is to leverage normalization techniques to auto- matically convert the non-standard words into the corresponding standard words (Aw et al., 2006; Cook and Stevenson, 2009; Pennell and Liu, 2011; Liu et al., 2012a; Li and Liu, 2014; Sonmez and Ozgur, 2014). Intuitively this will ease subsequent language processing modules. For example, if ‘2mr’ is converted to ‘tomorrow’, a text-to-speech system will know how to pronounce it, a part-of- speech (POS) tagger can label it correctly, and an information extraction system can identify it as a time expression. This normalization task has re- ceived an increasing attention in social media lan- guage processing. However, most of previous work on normaliza- tion assumed that they already knew which tokens are NSW that need normalization. Then differ- ent methods are applied only to these tokens. To our knowledge, Han and Baldwin (2011) is the only previous work which made a pilot research on NSW detection. One straight forward method to do this is to use a dictionary to classify a token …

Content truncated to 500 words