DuluthNLP at SemEval-2022 Task 7: Classifying Plausible Alternatives with Pre–trained ELECTRA

Proceedings of the 16th International Workshop on Semantic Evaluation (SemEval-2022), pages 1062 - 1066 July 14-15, 2022 ©2022 Association for Computational Linguistics DuluthNLP at SemEval-2022 Task 7: Classifying Plausible Alternatives with Pre–trained ELECTRA Samuel Akrah & Ted Pedersen Department of Computer Science University of Minnesota Duluth, MN 55812 USA � akrah001, tpederse � @d.umn.edu Abstract This paper describes the DuluthNLP system that participated in Task 7 of SemEval-2022 on Identifying Plausible Clarifications of Im- plicit and Underspecified Phrases in Instruc- tional Texts. Given an instructional text with an omitted token, the task requires models to clas- sify or rank the plausibility of potential fillers. To solve the task, we fine–tuned the models BERT, RoBERTa, and ELECTRA on training data where potential fillers are rated for plausi- bility. This is a challenging problem, as shown by BERT-based models achieving accuracy less than 45%. However, our ELECTRA model with tuned class weights on CrossEntropyLoss achieves an accuracy of 53.3% on the official evaluation test data, which ranks 6 out of the 8 total submissions for Subtask A. 1 Introduction Instructional texts (e.g., How To Guides) describe how to accomplish a given goal and are integral to our daily lives. One popular source is WikiHow1 which is an online platform that allows users to col- laborate to create and maintain such guides. This kind of documentation must be clear, and if it is not then this is a key reason that prompts revisions of underspecified instructions in WikiHow (Anthonio et al., 2020a). One important problem for NLP is to determine if a given instructional text is in need of clarifica- tion or revision. SemEval-2021 Task 7 (Roth et al., 2022) extends this problem by requiring models to score five possible fillers based on how well they can plausibly fit a given context. Task 7 includes two subtasks. Subtask A classifies the possible fillers as IMPLAUSIBLE, NEURAL, and PLAUS- BILE. Subtask B requires systems to rank the fillers on a scale of 1 to 5, where a higher score means more plausible. We only participated in Subtask A and used a variety of BERT-based methods. 1https://www.wikihow.com 2 Task Data The training, development and test data were sup- plied by the organizers of SemEval–2022 Task 7 (Roth et al., 2022). The dataset is based on the Wik- iHowToImprove Corpus (Anthonio et al., 2020b) which consists of edits of 2.5 million sentences from WikiHow. The authors show that edits are pri- marily made to clarify instructional texts and that the distinction between older and revised versions of sentences can be modelled computationally. Each instance in the data is divided into an Ar- ticle title, a Section header, and a Sentence nested between a Previous and Future context. Each in- stance also includes five potential fillers, each of which is annotated as IMPLAUSIBLE, NEUTRAL, or PLAUSIBLE, which serves as the basis of Sub- task A. There is a also a plausibility score of 1 to 5 which is the basis of Subtask B (which we did not …

Content truncated to 500 words