New Word Detection for Sentiment Analysis

Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics, pages 531–541, Baltimore, Maryland, USA, June 23-25 2014. c⃝2014 Association for Computational Linguistics New Word Detection for Sentiment Analysis Minlie Huang, Borui Ye*, Yichen Wang, Haiqiang Chen**, Junjun Cheng**, Xiaoyan Zhu State Key Lab. of Intelligent Technology and Systems, National Lab. for Information Science and Technology, Dept. of Computer Science and Technology, Tsinghua University, Beijing 100084, PR China *Dept. of Communication Engineering, Beijing University of Posts and Telecommunications **China Information Technology Security Evaluation Center aihuang@tsinghua.edu.cn Abstract Automatic extraction of new words is an indispensable precursor to many NLP tasks such as Chinese word segmentation, named entity extraction, and sentimen- t analysis. This paper aims at extract- ing new sentiment words from large-scale user-generated content. We propose a ful- ly unsupervised, purely data-driven frame- work for this purpose. We design statisti- cal measures respectively to quantify the utility of a lexical pattern and to measure the possibility of a word being a new word. The method is almost free of linguistic re- sources (except POS tags), and requires no elaborated linguistic rules. We also demonstrate how new sentiment word will benefit sentiment analysis. Experiment re- sults demonstrate the effectiveness of the proposed method. 1 Introduction New words on the Internet have been emerg- ing all the time, particularly in user-generated con- tent. Users like to update and share their infor- mation on social websites with their own language styles, among which new political/social/cultural words are constantly used. However, such new words have made many natural language processing tasks more challeng- ing. Automatic extraction of new words is indis- pensable to many tasks such as Chinese word seg- mentation, machine translation, named entity ex- traction, question answering, and sentiment analy- sis. New word detection is one of the most critical issues in Chinese word segmentation. Recent stud- ies (Sproat and Emerson, 2003) (Chen, 2003) have shown that more than 60% of word segmentation errors result from new words. Statistics show that more than 1000 new Chinese words appear every year (Thesaurus Research Center, 2003). These words are mostly domain-specific technical terms and time-sensitive political/social /cultural terms. Most of them are not yet correctly recognized by the segmentation algorithm, and remain as out of vocabulary (OOV) words. New word detection is also important for sen- timent analysis such as opinionated phrase ex- traction and polarity classification. A sentiment phrase with complete meaning should have a cor- rect boundary, however, characters in a new word may be broken up. For example, in a sentence " 表 演/ n 非 常/ adv 给/ v 力/ n（artists' perfor- mance is very impressive）" the two Chinese char- acters“给/v 力/n(cool; powerful)”should always be extracted together. In polarity classification, new words can be informative features for clas- sification models. In the previous example, "给 力(cool; powerful)" is a strong feature for clas- sification models while each single character is not. Adding new words as feature in classification models will improve the performance of polarity classification, as demonstrated …

Content truncated to 500 words