HeadlineCause: A Dataset of News Headlines for Detecting Causalities

Proceedings of the 13th Conference on Language Resources and Evaluation (LREC 2022), pages 6153–6161 Marseille, 20-25 June 2022 © European Language Resources Association (ELRA), licensed under CC-BY-NC-4.0 6153 HeadlineCause: A Dataset of News Headlines for Detecting Causalities Ilya Gusev, Alexey Tikhonov Moscow Institute of Physics and Technology, Independent researcher ilya.gusev@phystech.edu, altsoph@gmail.com Abstract Detecting implicit causal relations in texts is a task that requires both common sense and world knowledge. Existing datasets are focused either on commonsense causal reasoning or explicit causal relations. In this work, we present HeadlineCause, a dataset for detecting implicit causal relations between pairs of news headlines. The dataset includes over 5000 headline pairs from English news and over 9000 headline pairs from Russian news labeled through crowdsourcing. The pairs vary from totally unrelated or belonging to the same general topic to the ones including causation and refutation relations. We also present a set of models and experiments that demonstrates the dataset validity, including a multilingual XLM-RoBERTa based model for causality detection and a GPT-2 based model for possible effects prediction. Keywords: causality, dataset, news, headlines, BERT, GPT, multilingual 1. Introduction Causality is a crucial concept in many human activi- ties. Automatic inference of causal relations from texts is vital for any model attempting to analyze documents or predict future events. It is much more essential when we consider news. On the other hand, it is still a chal- lenging task for text understanding models, as it re- quires both common sense and world knowledge. From the practical sense, news aggregators are inter- ested in detecting causal relations. Firstly, they should understand what news documents refute others to build a relevant and rapid news feed, and a refutation is a special kind of causal relation. For example, almost every big accident has many death toll changes that re- fute each other. The rise of fake news also increases the need to detect refutations. Secondly, news aggrega- tors should be able to differentiate between news from different sources about the same event and news on the same topic but about different events. Cause-effect event pairs are hard negative samples for this task. There are several types of causal relations. In this work, we focus on implicit inter-sentence causal relations. It means that cause and effect are in different sentences, even in different texts in our case, and there are no ex- plicit linking words between them. This paper introduces a dataset of news headline pairs in English and Russian with causality labels obtained through crowdsourcing. We deliberately chose not to include texts of news documents in this dataset as al- most every headline contains only one fact and roughly corresponds to a notion of an event. Furthermore, us- ing headlines is much easier than detecting causalities between different parts of texts. For the same reasons, headlines were used in other works (Radinsky et al., 2012). Natural language understanding benchmarks such as SuperGLUE (Wang et al., 2019) and RussianSuper- GLUE (Shavrina et al., 2020) were introduced recently and are …

Content truncated to 500 words