Pheonix at SemEval-2020 Task 5: Masking the Labels Lubricates Models for Sequence Labeling

Proceedings of the 14th International Workshop on Semantic Evaluation, pages 677–682 Barcelona, Spain (Online), December 12, 2020. 677 Pheonix at SemEval-2020 Task 5: Masking the Labels Lubricates Models for Sequence Labeling Pouria Babvey, Dario Borrelli, Yutong Zhao, Carlo Lipizzi School of Systems and Enterprises Stevens Institute of Technology Hoboken, United States pbabvey,darioborrelli,yzhao102, clipizzi@stevens.edu Abstract This paper presents the deep-learning model that is submitted to the SemEval-2020 Task 5 competition: “Detecting Counterfactuals”. We participated in both Subtask1 and Subtask2. The model proposed in this paper ranked 2nd in Subtask2: “Detecting antecedent and consequence”. Our model approaches the task as a sequence labeling. The architecture is built on top of BERT; and a multi-head attention layer with label masking is used to benefit from the mutual information between nearby labels. Also, for prediction, a multi-stage algorithm is used in which the model finalize some predictions with higher certainty in each step and use them in the following. Our results show that masking the labels not only is an efficient regularization method but also improves the accuracy of the model compared with other alternatives like CRF. Label masking can be used as a regularization method in sequence labeling. Also, it improves the performance of the model by learning the specific patterns in the target variable. 1 Introduction Counterfactual statements describe events that did not actually happen or cannot happen, as well as the possible consequence. For example, in the sentence “If I was tall just like a giraffe, I was eating the leafs of those charming trees.”, the first clause is a counterfactual antecedent and the second clause is the consequent. To model counterfactual semantics and reasoning in natural language, SemEval-2020 Task 5 (Yang et al., 2020) aims to provide a benchmark for two basic problems: (1) detecting counterfactual statements as a binary classification task (2) detecting antecedent and consequent as a sequence labeling task. The introduced model in this paper is designed for the 2nd task. The model approaches the antecedent (ANT) and consequent (CONS) detection as a sequence labeling. Sequence labeling is the assignment of a categorical label to each member of a sequence. Common examples of a sequence labeling are part-of-speech tagging and named entity recognition. Although, sequence labeling can be treated as a set of independent classification tasks, considering the dependency of choices for nearby elements improves the accuracy to choose the globally best set of labels for the whole sequence. It is specially the case for some sequence labeling tasks like SQuAD (Rajpurkar et al., 2016), that the task is to find a set of consecutive tokens from a reading passage that respond to the specific question. Likewise, in counterfactual detection task, individual tokens in the same noun phrase, clause, or sentence are likely to get the same labels. In order to benefit from the mutual information between the labels of nearby tokens our model is built on top of BERT (Devlin et al., 2018). BERT benefits from two main features: (1) Pretraining: different forms of information about …

Content truncated to 500 words