niksss at SemEval-2022 Task7:Transformers for Grading the Clarifications on Instructional Texts

Proceedings of the 16th International Workshop on Semantic Evaluation (SemEval-2022), pages 1090 - 1093 July 14-15, 2022 ©2022 Association for Computational Linguistics niksss at SemEval-2022 Task 7: Transformers for Grading the Clarifications on Instructional Texts Nikhil Singh Manipal University Jaipur nikhil3198@gmail.com Abstract This paper describes the 9th place system de- scription for SemEval-2022 Task 7. The goal of this shared task was to develop computa- tional models to predict how plausible a clari- fication made on an instructional text is. This shared task was divided into two Subtasks A and B. We attempted to solve these using vari- ous transformers-based architecture under dif- ferent regime. We initially treated this as a text2text generation problem but comparing it with our recent approach we dropped it and treated this as a text-sequence classification and regression depending on the Subtask. 1 Introduction Instructional texts which are in the form of step- by-step instruction to achieve a particular goal are sometimes ambiguous to make out what is being talked about. To ensure that instructions describe clearly enough what steps must be followed, some clarifications are made in the places of ambiguity. This task (Roth et al., 2022) revolves around au- tomating the grading of a particular clarification made on the instructions into plausible implausible and neutral (SubTask A) and on a finer scale where it is required to rank potential clarifications from 1 to 5 (SubTask B). Previous work, related to this involves a Shared Task (Roth and Anthonio, 2021) which was a bi- nary classification task, in which systems had to predict whether a given sentence in context requires clarification or not. This shared task uses the same dataset that is the wikiHowToImprove dataset (An- thonio et al., 2020) but with some variations. In- stead of a binary classification task, this task is shaped as a cloze task in which, clarifications are presented as possible fillers and systems have to score how well each filler plausibly fits in a given context. A data instance can be seen in Figure 1. Seeing the performance of BERT over BiLSTMs in (Bhat et al., 2020), we decided to build upon that Figure 1: A data instance of Task 7 and explore the use of other transformers-based models. 2 System Overview Data Pre-Processing In order to convert the provided data into the form required for a text sequence classification model, we filled the blanks in the provided text with the provided potential fillers and associated their re- spective labels as shown in Figure 2. The resulting data were then divided into three parts training(80 % ), validating(10 % ), and testing(10 % ). 2.1 Subtask A For this task, we mainly experiment with two transformers-based models that differ fundamen- tally in the manner they were trained. Initially, we treated this as a text to text generation task using T5 (Raffel et al., 2019). The detailed steps involved in this experiment is present below. • The Article title, previous context, the sen- tence (with filler), and the follow-up …

Content truncated to 500 words