Noobs at Semeval-2021 Task 4: Masked Language Modeling for abstract answer prediction

Proceedings of the 15th International Workshop on Semantic Evaluation (SemEval-2021), pages 805–809 Bangkok, Thailand (online), August 5–6, 2021. ©2021 Association for Computational Linguistics 805 Noobs at Semeval-2021 Task 4: Masked Language Modeling for abstract answer prediction Shikhar Shukla , Sarthak Samsung Research Institute Bangalore-560037, India shikhar.00778@gmail.com,sarthak.j2709@gmail.com Karm Veer Arya ABV-Indian Institute of Information Technology & Management Gwalior-474015, India kvarya@iiitm.ac.in Abstract This paper presents the system developed by our team for Semeval 2021 Task 4: Reading Comprehension of Abstract Meaning. The aim of the task was to benchmark the NLP tech- niques in understanding the abstract concepts present in a passage, and then predict the miss- ing word in a human written summary of the passage. We trained a Roberta-Large model trained with a masked language modeling ob- jective. In cases where this model failed to predict one of the available options, another Roberta-Large model trained as a binary clas- sifier was used to predict correct and incorrect options. We used passage summary generated by Pegasus model and question as inputs. Our best solution was an ensemble of these 2 sys- tems. We achieved an accuracy of 86.22% on subtask 1 and 87.10% on subtask 2. 1 Introduction There has been a lot of research in evaluating the performance of machine learning models to iden- tify concrete concepts present in text and answer questions based on it (Hermann et al., 2015a). The organizers of ReCAM task at Semeval 2021 have provided a dataset to benchmark the models’ per- formance on understanding the abstract concepts in the text in English language. The models are required to predict the missing words in a human written summary of the passage. This can help assess if the models can accurately capture the im- portant concepts and meaning in the text. The paper is organized as follows: In Section 2, we give a background on the problem and method that has been used, in Section 3, we present the pro- posed system architecture, in Section 4, we present the hyperparameters analysis done on the system, in Section 5, we present the results of the proposed architecture along with other approaches taken, and in Section 6, we conclude the paper. 2 Background 2.1 Dataset The organizers provide two different datasets (Zheng et al., 2021) for two subtaks exploring two different definitions of abstractness (Spreen and Schulz, 1966; Changizi, 2008), imperceptibility and nonspecificity. Anything which can’t be perceived is described as an Imperceptible concept (Example: culture,economy etc.) (Spreen and Schulz, 1966; Coltheart, 1981; Turney et al., 2011). Nonspeci- ficity, as decribed by (Changizi, 2008), rather than looking at concrete things, focuses on generalizing the text (Example: hypernyms of words; vertebrate for whale). Subtask 3 explores the relationship between the two definitions. 2.2 Masked Language Modelling Masked Language Models have played an impor- tant role in BERT(Devlin et al., 2019) and sub- sequent transformer models’ success on different datasets. Masked Language Modelling is based on Cloze task (Taylor, 1953), which is described as filling the blanks in sentence using …

Content truncated to 500 words