LRG at SemEval-2021 Task 4: Improving Reading Comprehension with Abstract Words using Augmentation, Linguistic Features and Voting

Proceedings of the 15th International Workshop on Semantic Evaluation (SemEval-2021), pages 189–198 Bangkok, Thailand (online), August 5–6, 2021. ©2021 Association for Computational Linguistics 189 LRG at SemEval-2021 Task 4: Improving Reading Comprehension with Abstract Words using Augmentation, Linguistic Features and Voting Abheesht Sharma∗ Dept. of CS&IS BITS Pilani, Goa Campus f20171014@goa.bits-pilani.ac.in Harshit Pandey∗ Dept. of Computer Science Pune University hp2pandey1@gmail.com Gunjan Chhablani∗ Dept. of CS&IS BITS Pilani, Goa Campus chhablani.gunjan@gmail.com Yash Bhartia Dept. of CS&IS BITS Pilani, Goa Campus f20190151@goa.bits-pilani.ac.in Tirtharaj Dash APP Centre for AI Research BITS Pilani, Goa Campus tirtharaj@goa.bits-pilani.ac.in Abstract In this article, we present our methodologies for SemEval-2021 Task-4: Reading Compre- hension of Abstract Meaning. Given a fill-in- the-blank-type question and a corresponding context, the task is to predict the most suitable word from a list of 5 options. There are three sub-tasks within this task: Imperceptibility (subtask-I), Non-Specificity (subtask-II), and Intersection (subtask-III). We use encoders of transformers-based models pre-trained on the masked language modelling (MLM) task to build our Fill-in-the-blank (FitB) models. Moreover, to model imperceptibility, we de- fine certain linguistic features, and to model non-specificity, we leverage information from hypernyms and hyponyms provided by a lexi- cal database. Specifically, for non-specificity, we try out augmentation techniques, and other statistical techniques. We also propose vari- ants, namely Chunk Voting and Max Context, to take care of input length restrictions for BERT, etc. Additionally, we perform a thor- ough ablation study, and use Integrated Gradi- ents to explain our predictions on a few sam- ples. Our best submissions achieve accura- cies of 75.31% and 77.84%, on the test sets for subtask-I and subtask-II, respectively. For subtask-III, we achieve accuracies of 65.64% and 62.27%. The code is available here. 1 Introduction A very common assessment in schools is question- answering based on a given “comprehension pas- sage”. Students are given a comprehension passage, from which they are supposed to glean necessary information, and answer short questions (such as fill-in-the-blanks-type question) based on what they have garnered from the given passage. While trying to find the most appropriate word for the blank, the children look at the words surrounding the blank ∗ Equal contribution. Author ordering determined by coin flip. (“context”). The word should be such that when the word fills the blank, the sentence makes sense and it is grammatically correct. Inspired by this, and perhaps, after the enormous success of Transform- ers (Vaswani et al., 2017), researchers at Google came up with a large number of “pretraining tasks” and built knowledge-heavy language models which could be fine-tuned on various natural language processing (NLP) downstream tasks. One of the earlier pretraining tasks was “Masked Language Modelling (MLM)”, one of the two pretraining tasks of the breakthrough model, BERT (Devlin et al., 2019). The approach here was similar to how kids are taught language at school: some tokens in the text were randomly “masked” and the model was trained to predict these masked tokens. SemEval-2021 Task-4 (Zheng et al., 2021) fo- cuses on a similar idea. Every sample …

Content truncated to 500 words