Drop-out Conditional Random Fields for Twitter with Huge Mined Gazetteer

Proceedings of NAACL-HLT 2016, pages 282–288, San Diego, California, June 12-17, 2016. c⃝2016 Association for Computational Linguistics Drop-out Conditional Random Fields for Twitter with Huge Mined Gazetteer Eunsuk Yang‡§ Young-Bum Kim†§ Ruhi Sarikaya† Yu-Seop Kim‡ †Microsoft Corporation, Redmond, WA ‡Hallym University, South Korea esyang219@gmail.com {ybkim, ruhi.sarikaya}@microsoft.com yskim01@hallym.ac.kr Abstract In named entity recognition task especially for massive data like Twitter, having a large amount of high quality gazetteers can allevi- ate the problem of training data scarcity. One could collect large gazetteers from knowl- edge graph and phrase embeddings to obtain high coverage of gazetteers. However, large gazetteers cause a side-effect called “feature under-training”, where the gazetteer features overwhelm the context features. To resolve this problem, we propose the dropout condi- tional random fields, which decrease the influ- ence of gazetteer features with a high weight. Our experiments on named entity recognition with Twitter data lead to higher F1 score of 69.38%, about 4% better than the strong base- line presented in Smith and Osborne (2006). 1 Introduction Nowadays, people are generating tremendous amount of information on social websites. For ex- ample, more than 200 million tweets are generated everyday on Twitter (Ritter et al., 2011). Twitter has become a key news source, in addition to standard news channels. As such, social scientists are start- ing to pay attention to it in recent years (Bollen et al., 2011; Chung and Mustafaraj, 2011; Xu et al., 2014; Calvin et al., 2015; Baldwin et al., 2015; Bell- more et al., 2015). The traditional machine learned modeling approaches trained with small and clean general text, such as news articles, perform poorly when applied to tweets, because tweets are struc- turally very different from general text. Thus, it § Both authors contributed equally. is necessary to build new models for Twitter. One could label a reasonable size of tweets to train a model for a natural language processing (NLP) ap- plication. The problem is that it is very expensive to refresh the annotated data to keep the model up- to-date, because users generate tweets in a unprece- dented rate (Hachman, 2011). An obvious solution to the problem is to de- velop methods of utilizing a large amount of un- labeled data. One way is to induce word embed- dings in a real-valued vector space from a large num- ber of tweets (Kim et al., 2015a; Mikolov et al., 2013; Pennington et al., 2014). It is shown that the task-specific embeddings induced on tweets pro- vide more powerful than those created from out-of- domain texts (Owoputi et al., 2012; Anastasakos et al., 2014). Another method is to build the task-specific gazetteers. Task-specific gazetteers make the mod- els more general and increase their coverage for un- seen events. They have been proven to be useful on a number of tasks (Smith and Osborne, 2006; Li et al., 2009; Liu and Sarikaya, 2014; Kim et al., 2015b; Kim et al., 2015c). Since gazetteers can improve modeling performance, here we more focus on how to use gazetteer more effectively. …

Content truncated to 500 words