JBNU-CCLab at SemEval-2022 Task 7: DeBERTa for Identifying Plausible Clarifications in Instructional Texts

Proceedings of the 16th International Workshop on Semantic Evaluation (SemEval-2022), pages 1050 - 1055 July 14-15, 2022 ©2022 Association for Computational Linguistics JBNU-CCLab at SemEval-2022 Task 7: DeBERTa for Identifying Plausible Clarifications in Instructional Texts Daewook Kang, Sung-Min Lee, Eunhwan Park, Seung-Hoon Na Computer Science and Engineering, Jeonbuk National University, South Korea {dwkng, cap1232, judepark, nash}@jbnu.ac.kr Abstract In this study, we examine the ability of con- textualized representations of pretrained lan- guage model to distinguish whether sequences from instructional articles are plausible or im- plausible. Towards this end, we compare the BERT, RoBERTa, and DeBERTa models using simple classifiers based on the sentence repre- sentations of the [CLS] tokens and perform a detailed analysis by visualizing the representa- tions of the [CLS] tokens of the models. In the experimental results of Subtask A: Multi-Class Classification, DeBERTa exhibits the best per- formance and produces a more distinguishable representation across different labels. Submit- ting an ensemble of 10 DeBERTa-based mod- els, our final system achieves an accuracy of 61.4% and is ranked fifth out of models sub- mitted by eight teams. Further in-depth results suggest that the abilities of pretrained language models for the plausibility detection task are more strongly affected by their model struc- tures or attention designs than by their model sizes. 1 Introduction WikiHow1 is the largest how-to website with more than 300,000 articles and over 2.5M regis- tered users that help user improve their knowledge of specific areas. However, these instructional arti- cles have grammatical errors and ambiguous con- tent that cause misunderstandings. To enhance the clarity of instructional texts, clarification is required as a revision that makes implicit elements explicit, resolves ambiguities, or replaces under- specified phrases with a clearer and more precise expressions. SemEval-2022 Task 7 (Roth et al., 2022) eval- uates the ability of an NLP system to distinguish between plausible and implausible clarifications of an instruction. The task is formulated as a CLOZE task in which clarification is presented as a filler in 1https://www.wikihow.com/Main-Page a blanked sentence. Given a context with a filler option (i.e., a sentence X and filler option O), the system should determine the plausibility of a se- quence, that is whether a sequence is “plausible,” “neutral,” and “implausible.” Our work is motivated by the recent successes of pretrained language models such as BERT (De- vlin et al., 2019), RoBERTa (Liu et al., 2019), and DeBERTa (He et al., 2021a,b), which have effectively induced contextualized representations, achieving remarkable fine-tuning performance on downstream tasks. To explore the plausibility detection of clari- fications in SemEval-2022 Task 7, we compare the plausibility detection abilities of three pre- trained language models (BERT, RoBERTa, and DeBERTa) in Subtask A, Multi-Class Classifica- tion. Our main observations of the development set are highlighted as follows. • Among the three models, DeBERTa exhibits the best performance in the plausibility classi- fication task2. • Visualization analysis of the representations of the [CLS] tokens of BERT, RoBERTa, and DeBERTa, reveals that the best distinguish- able representations among different classes are achieved …

Content truncated to 500 words