TWEETQA: A Social Media Focused Question Answering Dataset

Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 5020–5031 Florence, Italy, July 28 - August 2, 2019. c⃝2019 Association for Computational Linguistics 5020 TWEETQA: A Social Media Focused Question Answering Dataset Wenhan Xiong†, Jiawei Wu†, Hong Wang†, Vivek Kulkarni†, Mo Yu∗, Shiyu Chang∗, Xiaoxiao Guo∗, William Yang Wang† † University of California, Santa Barbara ∗ IBM Research {xwhan, william}@cs.ucsb.edu, yum@us.ibm.com, {shiyu.chang, xiaoxiao.guo}@ibm.com Abstract With social media becoming increasingly pop- ular on which lots of news and real-time events are reported, developing automated question answering systems is critical to the effective- ness of many applications that rely on real- time knowledge. While previous datasets have concentrated on question answering (QA) for formal text like news and Wikipedia, we present the first large-scale dataset for QA over social media data. To ensure that the tweets we collected are useful, we only gather tweets used by journalists to write news articles. We then ask human annotators to write questions and answers upon these tweets. Unlike other QA datasets like SQuAD in which the answers are extractive, we allow the answers to be ab- stractive. We show that two recently proposed neural models that perform well on formal texts are limited in their performance when ap- plied to our dataset. In addition, even the fine- tuned BERT model is still lagging behind hu- man performance with a large margin. Our re- sults thus point to the need of improved QA systems targeting social media text. 1 1 Introduction Social media is now becoming an important real- time information source, especially during nat- ural disasters and emergencies. It is now very common for traditional news media to frequently probe users and resort to social media platforms to obtain real-time developments of events. Ac- cording to a recent survey by Pew Research Cen- ter2, in 2017, more than two-thirds of Americans read some of their news on social media. Even for American people who are 50 or older, 55% of them report getting news from social media, 1The Dataset can be found at https://tweetqa. github.io/. 2http://www.journalism.org/2017/09/07/news-use- across-social-media-platforms-2017/ Passage: Oh man just read about Paul Walk- ers death. So young. Ugggh makes me sick especially when it’s caused by an accident. God bless his soul. – Jay Sean (@jaysean) December 1, 2013 Q: why is sean torn over the actor’s death? A: walker was young Table 1: An example showing challenges of TWEETQA. Note the highly informal nature of the text and the presence of social media specific text like user- names which need to be comprehended to accurately answer the question. which is 10% points higher than the number in 2016. Among all major social media sites, Twit- ter is most frequently used as a news source, with 74% of its users obtaining their news from Twitter. All these statistical facts suggest that understand- ing user-generated noisy social media text from Twitter is a significant task. In recent years, while several tools for core nat- ural language understanding tasks involving syn- …

Content truncated to 500 words