CNRL at SemEval-2020 Task 5: Modelling Causal Reasoning in Language with Multi-Head Self-Attention Weights Based Counterfactual Detection

Proceedings of the 14th International Workshop on Semantic Evaluation, pages 451–457 Barcelona, Spain (Online), December 12, 2020. 451 CNRL at SemEval-2020 Task 5: Modelling Causal Reasoning in Language with Multi-Head Self-Attention Weights based Counterfactual Detection Rajaswa Patil1,2 and Veeky Baths1,3 1Cognitive Neuroscience Lab, BITS Goa 2Department of Electrical & Electronics Engineering 3Department of Biological Sciences BITS Pilani K. K. Birla Goa Campus, India {f20170334, veeky}@goa.bits-pilani.ac.in Abstract In this paper, we describe an approach for modelling causal reasoning in natural language by detecting counterfactuals in text using multi-head self-attention weights. We use pre-trained transformer models to extract contextual embeddings and self-attention weights from the text. We show the use of convolutional layers to extract task-specific features from these self-attention weights. Further, we describe a fine-tuning approach with a common base model for knowledge sharing between the two closely related sub-tasks for counterfactual detection. We analyze and compare the performance of various transformer models in our experiments. Finally, we perform a qualitative analysis with the multi-head self-attention weights to interpret our models’ dynamics. 1 Introduction Causal reasoning is a process of detecting cause-effect relationships and is increasingly being used in artificial intelligence for improving generalization and interpretability. Modelling causal reasoning in language involves detecting such cause-effect relationships from natural language texts. A cause-effect relationship can be modelled as: Event A causes Event B. Counterfactuals describe events counter to facts and hence naturally involve common sense and causal reasoning. A counterfactual can be modelled as a cause-effect relationship of the form: Event A could have caused Event B (Event A did not occur). SemEval-2020 Task-5 (Yang et al., 2020) consists of two independent sub-tasks: a binary-classification task for detecting counterfactual statements and a span-detection task for detecting antecedent (cause) - consequent (effect) spans of given counterfactual statements (Table 1). In this work, we use multi-head self-attention weights from pre-trained transformer models (Vaswani et al., 2017) to capture the semantic interactions between the tokens of given text with respect to causal relations. We use a fine-tuning approach with a common base model for knowledge sharing between these two closely related sub-tasks. The code for this work is made publicly available as a GitHub repository.1 2 Background Early work on causal reasoning and related tasks in natural language was based on various statistical and linguistic approaches (Asghar, 2016). Recent work for causal reasoning related tasks involves deep learning based approaches. Causal reasoning can be achieved through extraction of cause-effect relations with CRF and LSTM based sequence labelling tasks (Dasgupta et al., 2018). Counterfactuals can contain implicit causal relations (Table 1). Using multi-head self-attention at word level can help capture such implicit causal relations effectively (Liang et al., 2019). Current benchmarks for modelling causal reasoning involves question-answering tasks (Gordon et al., 2012). Using pre-trained transformer models have been effective on such tasks (Sap et al., 2019). Self-attention weights from such transformer models are usually structured in a 3-dimensional matrix. Using convolutional neural networks with these self-attention weight matrices can be helpful for extracting semantic features for …

Content truncated to 500 words