TSix: A Human-involved-creation Dataset for Tweet Summarization

TSix: A Human-involved-creation Dataset for Tweet Summarization Minh-Tien Nguyen1,2, Dac Viet Lai1, Huy-Tien Nguyen1, Le-Minh Nguyen1 1Japan Advanced Institute of Science and Technology, 1-1 Asahidai, Nomi, Ishikawa, 923-1292, Japan. {tiennm, vietld, ntienhuy, nguyenml}@jaist.ac.jp 2 Hung Yen University of Technology and Education, Hung Yen, Vietnam. Abstract We present a new dataset for tweet summarization. The dataset includes six events collected from Twitter from October 10 to November 9, 2016. Our dataset features two prominent properties. Firstly, human-annotated gold-standard references allow to correctly evaluate extractive summarization methods. Secondly, tweets are assigned into sub-topics divided by consecutive days, which facilitate incremental tweet stream summarization methods. To reveal the potential usefulness of our dataset, we compare several well-known summarization methods. Experimental results indicate that among extractive approaches, hybrid term frequency – document term frequency obtains competitive results in term of ROUGE-scores. The analysis also shows that polarity is an implicit fac- tor of tweets in our dataset, suggesting that it can be exploited as a component besides tweet content quality in the summarization process. Keywords: Tweet summarization, hybrid TF-IDF, dataset, corpus, annotation. 1. Introduction The growth of micro-blogging services such as Twitter en- courages users sharing their viewpoints regarding an event. For example, users following US Election can post their tweets (short messages with a maximum of 140 char- acters) on their timelines. After posting, their friends can immediately update new information about this event. Those who are out of their networks can also track the event by using the keyword-search function provided by Twitter. However, search results are usually overwhelming due to millions of returned tweets, which span for weeks. Even if the filter is enabled, digging a large number of tweets for interesting contents would be a nightmare due to their noise. These demand a topic-driven system extracting high- quality tweets for user interests. The bottleneck of social short-text summarization is the shortage of standard datasets for evaluating summariza- tion methods whereas well-known DUC datasets have been freely published for document summarization. For tweet summarization, authors usually create their data. For exam- ple, although (Shou et al., 2013) released a dataset includ- ing events for evaluating their method, the dataset is now inaccessible. (Imran et al., 2014) published a dataset for disaster response during the Joplin tornado collected from Twitter. Although this dataset contains more than 230,000 tweets, its lack of references challenges the evaluation. This paper leverages tweet summarization by introducing a new dataset including six events collected from Twitter. The involvement of humans in creating gold-standard ref- erences facilitates the evaluation. To show the potential us- ability of our dataset, we employ hybrid term frequency – inverse document frequency (TF-IDF) for extracting im- portant tweets. Experimental results show that the hy- brid model achieves competitive ROUGE scores over base- lines. We also further analyze the polarity aspect of tweets. The analysis shows that extracted tweets tend to be non- sentiment. The dataset can be publicly accessible.1 1https://goo.gl/kXBof9 2. Summarization This section presents our proposal in creating the dataset in two …

Content truncated to 500 words