CS-NET at SemEval-2020 Task 4: Siamese BERT for ComVE

Proceedings of the 14th International Workshop on Semantic Evaluation, pages 501–506 Barcelona, Spain (Online), December 12, 2020. 501 CS-NET at SemEval-2020 Task 4: Siamese BERT for ComVE Soumya Ranjan Dash∗ Sandeep Routray∗ Prateek Varshney∗ Ashutosh Modi Indian Institute of Technology Kanpur (IITK) {soumyard, sroutray}@iitk.ac.in, varshney@cse.iitk.ac.in ashutoshm@cse.iitk.ac.in Abstract In this paper, we describe our system for Task 4 of SemEval 2020, which involves differentiating between natural language statements that confirm to common sense and those that do not. The organizers propose three subtasks - first, selecting between two sentences, the one which is against common sense. Second, identifying the most crucial reason why a statement does not make sense. Third, generating novel reasons for explaining the against common sense statement. Out of the three subtasks, this paper reports the system description of subtask A and subtask B. This paper proposes a model based on transformer neural network architecture for addressing the subtasks. The novelty in work lies in the architecture design, which handles the logical implication of contradicting statements and simultaneous information extraction from both sentences. We use a parallel instance of transformers, which is responsible for a boost in the performance. We achieved an accuracy of 94.8% in subtask A and 89% in subtask B on the test set. 1 Introduction Incorporating common sense in natural language understanding systems and evaluating whether a system has sense-making capability remains a fundamental question in the natural language processing field (Modi, 2017; Modi, 2016; Modi and Titov, 2014). One important difference between human and machine text understanding lies in the fact that humans have access to commonsense knowledge while processing text, which helps them to draw inferences about facts that are not mentioned in a text, but that is assumed to be common ground (Modi et al., 2017). For a computer system, inferring unmentioned facts is a non-trivial challenge (Ostermann et al., 2018a). For our problem, we have proposed methods to include common sense in the validation and reasoning paradigm (Wang et al., 2019). Task 4 of semeval 2020 (Wang et al., 2020) is a common-sense validation and explanation task. It consists of classifying against common sense sentences from sentences that make sense. Figure 1 shows examples from subtask A and subtask B. In subtask A, clearly sentence 1 is against common sense. Subtask B contains three options for reasons to explain why sentence 1 is against common sense. As orange juice does not taste good on cereal, but milk does, sentence 1 makes less sense than sentence 2. We use the generated embedding from transformer based encoders like BERT (Devlin et al., 2018), RoBERTa (Liu et al., 2019), AlBERT (Lan et al., 2019) which capture in-context semantic information and design a siamese architecture to extract the relational information between the sentences. The implementation for our system is made available via Github1. 2 Background 2.1 Problem Definition The task is to develop a system that can differentiate natural language statements that make sense from those that do not. The organizers present three subtasks …

Content truncated to 500 words