Nowruz at SemEval-2022 Task 7: Tackling Cloze Tests with Transformers and Ordinal Regression

Proceedings of the 16th International Workshop on Semantic Evaluation (SemEval-2022), pages 1071 - 1077 July 14-15, 2022 ©2022 Association for Computational Linguistics Nowruz at SemEval-2022 Task 7: Tackling Cloze Tests with Transformers and Ordinal Regression Mohammadmahdi Nouriborji NODET, Iran∗ mohammadmahdinoori70@gmail.com Omid Rohanian Department of Engineering Science University of Oxford Oxford, UK omid.rohanian@eng.ox.ac.uk David Clifton Department of Engineering Science University of Oxford Oxford, UK david.clifton@eng.ox.ac.uk Abstract This paper outlines the system using which team Nowruz participated in SemEval 2022 Task 7 “Identifying Plausible Clarifications of Implicit and Underspecified Phrases” for both subtasks A and B (Roth et al., 2022). Using a pre-trained transformer as a backbone, the model targeted the task of multi-task classifica- tion and ranking in the context of finding the best fillers for a cloze task related to instruc- tional texts on the website Wikihow. The system employed a combination of two ordinal regression components to tackle this task in a multi-task learning scenario. According to the official leaderboard of the shared task, this system was ranked 4th in both classification and ranking subtasks out of 21 participating teams. With additional exper- iments, the models have since been further optimised. The code used in the experiments is going to be freely available at https:// github.com/mohammadmahdinoori/ Nowruz-at-SemEval-2022-Task-7. 1 Introduction Oxford dictionary defines cloze test as “a test of readability or comprehension in which a person is required to supply words which have been delib- erately omitted from a passage” (Oxford Univer- sity Press, 2022). In the context of NLP, a cloze format task is one in which the context is one or more sentences with masked spans and the model is expected to predict a suitable filler for each span. Cloze-format datasets have become popular in NLP recently as they are relatively easy to create auto- matically and provide high quality resources for model training (Rogers et al., 2021). 0National Organization For Development of Exceptional Talents SemEval 2022 task 7 is framed as a cloze task in which the goal is to rank or classify fillers within a given context based on their suitability. The texts are taken from actual articles on an instructional website and the masked spans are placed at loca- tions of edits made by users. We participated in this shared task in both ranking and classification parts and developed a transformer-based model that utilises both classification and regression compo- nents at the top layer. The code and the data used in these experiments are publicly available. 2 Related Work Cloze tasks have been a subject of interest in Natu- ral Language Understanding (NLU) in recent years, especially within the context of reading compre- hension, story understanding, and summarisation (Deutsch and Roth, 2019; Sharma et al., 2018; Mostafazadeh et al., 2016). There is evidence that cloze tasks can be used to effectively pretrain or finetune language models in order to perform few shot learning (Schick and Schütze, 2021; Liu et al., 2021). WikiHow is a community-edited open domain repository that hosts how-to articles on a variety …

Content truncated to 500 words