ReCAM@IITK at SemEval-2021 Task 4: BERT and ALBERT based Ensemble for Abstract Word Prediction

Proceedings of the 15th International Workshop on Semantic Evaluation (SemEval-2021), pages 175–182 Bangkok, Thailand (online), August 5–6, 2021. ©2021 Association for Computational Linguistics 175 ReCAM@IITK at SemEval-2021 Task 4: BERT and ALBERT based Ensemble for Abstract Word Prediction Abhishek Mittal Ashutosh Modi Indian Institute of Technology Kanpur (IIT Kanpur) amittal@iitk.ac.in ashutoshm@cse.iitk.ac.in Abstract This paper describes our system for Task 4 of SemEval-2021: Reading Comprehen- sion of Abstract Meaning (ReCAM). We participated in all subtasks where the main goal was to predict an abstract word miss- ing from a statement. We fine-tuned the pre-trained masked language models namely BERT and ALBERT and used an ensemble of these as our submitted system on Sub- task 1 (ReCAM-Imperceptibility) and Sub- task 2 (ReCAM-Nonspecificity). For Sub- task 3 (ReCAM-Intersection), we submitted the ALBERT model as it gives the best results. We tried multiple approaches and found that Masked Language Modeling(MLM) based ap- proach works the best. 1 Introduction Computers’ ability to understand, represent, and express text with abstract meaning is a fundamen- tal problem towards achieving true natural lan- guage understanding. In past decades, significant advancement has been achieved in representation learning. SemEval-2021 Task 4 : Reading Com- prehension of Abstract Meaning (ReCAM) (Zheng et al., 2021) explores the ability of machines to understand abstract concepts and proposes to pre- dict abstract words just as humans do while writing article summaries. In the shared task, text pas- sages are provided to read and understand abstract meaning. It consists of three subtasks where the first two subtasks are based on two different defi- nitions of abstractness 1) Imperceptibility (Spreen and Schulz, 1966) and 2) Non-specificity (Changizi et al., 2008) and the third subtask discusses their intersection. Many cloze-style reading comprehension datasets like CNN/Daily Mail (Hermann et al., 2015) and Children’s Book Test (CBTest) dataset (Hill et al., 2016) and models (Dhingra et al., 2016; Munkhdalai and Yu, 2017) similar to this task exist, where a missing word has to be inferred. However, these previous datasets and models have mostly focused on inferring concrete words or concepts like named entities, but this task moves the focus from concreteness to abstractness of words in reading comprehension. This can prove to be quite useful for current ongoing research in the field of abstractive summarization. We participated in all the three subtasks. We mainly used an ensemble of BERT and ALBERT as our final model for submission on subtasks 1 and 2. We were ranked 13th on Subtask 1 and 11th on Subtask 2. We submitted the ALBERT model on Subtask 3. All of our code is made publicly avail- able on Github1. We approached this task in two ways. One is a Multiple Choice Question answer- ing (MCQ) based approach and other a Masked Language Modeling (MLM) approach. Through experiments, we concluded that such tasks are best addressed using a masked language model. The rest of the paper is organised as follows. Section 2 describes the problem statement formally and also gives a brief description of the dataset …

Content truncated to 500 words