ForecastQA: A Question Answering Challenge for Event Forecasting with Temporal Text Data

Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing, pages 4636–4650 August 1–6, 2021. ©2021 Association for Computational Linguistics 4636 FORECASTQA: A Question Answering Challenge for Event Forecasting with Temporal Text Data Woojeong Jin1 Rahul Khanna1 Suji Kim1 Dong-Ho Lee1 Fred Morstatter2 Aram Galstyan2 Xiang Ren1 2 1Department of Computer Science, University of Southern California 2Information Sciences Institute, University of Southern California {woojeong.jin, rahulkha, sujikim, donghole, xiangren}@usc.edu, {fredmors, galstyan}@isi.edu Abstract Event forecasting is a challenging, yet impor- tant task, as humans seek to constantly plan for the future. Existing automated forecast- ing studies rely mostly on structured data, such as time-series or event-based knowledge graphs, to help predict future events. In this work, we aim to formulate a task, construct a dataset, and provide benchmarks for de- veloping methods for event forecasting with large volumes of unstructured text data. To simulate the forecasting scenario on tempo- ral news documents, we formulate the prob- lem as a restricted-domain, multiple-choice, question-answering (QA) task. Unlike exist- ing QA tasks, our task limits accessible in- formation, and thus a model has to make a forecasting judgement. To showcase the use- fulness of this task formulation, we introduce FORECASTQA, a question-answering dataset consisting of 10,392 event forecasting ques- tions, which have been collected and verified via crowdsourcing efforts. We present our experiments on FORECASTQA using BERT- based models and find that our best model achieves 61.0% accuracy on the dataset, which still lags behind human performance by about 19%. We hope FORECASTQA will support fu- ture research efforts in bridging this gap.1 1 Introduction Forecasting globally significant events, such as outcomes of policy decisions, civil unrest, or the economic ramifications of global pandemics, is a consequential but arduous problem. In recent years there have been significant advances in apply- ing machine learning (e.g., time-series prediction methods) to generate forecasts for various types of events including conflict zones (Schutte, 2017), duration of insurgency (Pilster and B¨ohmelt, 2014), civil unrest (Ramakrishnan et al., 2014a) and ter- rorist events (Raghavan et al., 2013). 1https://inklab.usc.edu/ForecastQA/ A) South Korea [0.41] B) Syria [0.28] C) South Africa [0.15] D) Portugal [0.16] Yes [0.38] / No [0.62] Q: Who will drop Japan as a trading partner in August 2019? (1/1/19) Apart from the fact of being one another’s closest neighbors, the people of South Korea and Japan have a remarkable amount in common. Economically, they are among one another’s biggest trading partners. And yet, time and again, relations between Seoul and Tokyo are marked, not by mutual support and co-operation but by anger, reproach and exasperation. Q: Will primary schools in Europe admit non-vaccinated children around September 2019? (3/8/18) Public officials and health experts had given several warnings: Do not allow a student in school if they had not been vaccinated against measles. (6/27/19) Fines for parents refusing measles jab. Parents will be fined up to € 2,500 if they don’t vaccinate their children against measles under draft legislation …

Content truncated to 500 words