Yseop at SemEval-2020 Task 5: Cascaded BERT Language Model for Counterfactual Statement Analysis

Proceedings of the 14th International Workshop on Semantic Evaluation, pages 468–478 Barcelona, Spain (Online), December 12, 2020. 468 Yseop at SemEval-2020 Task 5: Cascaded BERT Language Model for Counterfactual Statement Analysis Hanna Abi Akl Yseop habi-akl@yseop.com Dominique Mariko Yseop dmariko@yseop.com Estelle Labidurie Yseop elabidurie@yseop.com Abstract In this paper, we explore strategies to detect and evaluate counterfactual sentences. We describe our system for SemEval-2020 Task 5: Modeling Causal Reasoning in Language: Detecting Counterfactuals. We use a BERT base model for the classification task and build a hybrid BERT Multi-Layer Perceptron system to handle the sequence identification task. Our experiments show that while introducing syntactic and semantic features does little in improving the system in the classification task, using these types of features as cascaded linear inputs to fine-tune the sequence-delimiting ability of the model ensures it outperforms other similar-purpose complex systems like BiLSTM-CRF in the second task. Our system achieves an F1 score of 85.00% in Task 1 and 83.90% in Task 2. 1 Introduction A counterfactual can be defined as something that is contrary to the truth or that did not actually occur. It refers to an event that did not or cannot happen, as well as the possible consequences if it had happened. In the sentence ”If dogs had no ears, they could not hear”, the statement ”if dogs had no ears” is an example of a counterfactual because dogs do have ears. Task 5 of SemEval-2020 (Yang et al., 2020) focuses on identifying these specific sentence types amongst sentences delivering close semantic similarities. This implies understanding and disambiguating the causal link between two sentence fragments. We approached this task as an opportunity to test the effectiveness of disambiguating at a grammatical level against traditional baseline systems. This paper describes a parallel approach in deriving some meaning from text to leverage the influence of context and relevance of structure in recognizing counter- factual statements. Specifically, we will explore how many expressions of such statements a mapping of grammatical types can cover before falling short to high-performing models, most notably BERT (Devlin et al., 2018), which performed well on both tasks with an F1 score of 85.00% on Task 1 and 83.90 % on Task 2. 2 Related Work Although the task of detecting counterfactuals is relatively new, (Son et al., 2017) proposes using modal logic to form rule-based determination methods from social media posts. These methods are supplemented by a statistical classifier (Linear SVM) that is retrained to tackle more challenging counterfactual forms. Previous work on causal identification by (Levin and Hovav, 1994) studied the contribution of verbs in the determination of causal relations. By analyzing closely different formulations, they concluded that similarities in meaning can be derived from very different syntactic structures As for causality relation extraction, different deep learning systems built on the success of neural-based models have been proposed. The linguistically informed CNN model (Dasgupta et al., 2018) leverages the use of word embeddings and other linguistic features to detect causal patterns and outperforms rule-based classification. …

Content truncated to 500 words