X-PuDu at SemEval-2022 Task 7: A Replaced Token Detection Task Pre-trained Model with Pattern-aware Ensembling for Identifying Plausible Clarifications

Proceedings of the 16th International Workshop on Semantic Evaluation (SemEval-2022), pages 1078 - 1083 July 14-15, 2022 ©2022 Association for Computational Linguistics X-PuDu at SemEval-2022 Task 7: A Replaced Token Detection Task Pre-trained Model with Pattern-aware Ensembling for Identifying Plausible Clarifications Junyuan Shang1, Shuohuan Wang1, Yu Sun1, Yanjun Yu2, Yue Zhou2, Li Xiang2 and Guixiu Yang2 1 Baidu Inc., China 2 Shanghai Pudong Development Bank, China {shangjunyuan, wangshuohuan, sunyu02}@baidu.com {yuyj6, zhouy93, xiangl3, yanggx1}@spdb.com.cn Abstract This paper describes our winning system on SemEval 2022 Task 7: Identifying Plausi- ble Clarifications of Implicit and Underspec- ified Phrases in Instructional Texts. A re- placed token detection pre-trained model is uti- lized with minorly different task-specific heads for SubTask-A: Multi-class Classification and SubTask-B: Ranking. Incorporating a pattern- aware ensemble method, our system achieves a 68.90% accuracy score and 0.8070 spear- man’s rank correlation score surpassing the 2nd place with a large margin by 2.7 and 2.2 per- cent points for SubTask-A and SubTask-B, re- spectively. Our approach is simple and easy to implement, and we conducted ablation studies and qualitative and quantitative analyses for the working strategies used in our system. 1 Introduction The Internet’s ever-increasing size has made it easy to find instructional texts such as articles in wik- iHow1, on almost any topic or activity. Regular revisions of these how-to manuals are necessary to ensure that instructions communicate the pro- cedures required to attain a certain goal precisely. This shared task is introduced by Roth et al. (2022), whose intention is to find ways to improve instruc- tional texts, evaluate to what extent current NLP systems are able to handle implicit, ambiguous, and underspecified language, and go beyond the surface form of a text and take multiple plausible interpretations into account. Thus, the proposed NLP systems should be capable of distinguishing between plausible and implausible clarifications of an instruction shown in Figure. 1. The shared task consists of two subtasks: • SubTask-A: Multi-Class Classification. The goal is to predict a class label (IMPLAUSI- 1https://www.wikihow.com/Main-Page How to Print a Word Documen 10 Second Sumarray … (...) 2. Open or create a ______ Word document. 3. user (2.75) page (2.0) Microsoft (5.0) software (2.0) forum (1.25) Title Sec. Header Prev. Sent. Target Sent. Follow-up Sent. Clarifications (fillers) IMPLICIT REFERENCE Pattern Figure 1: An example randomly chosen from the dev set. Each sample is associated with five clarifications labeled ( PLAUSIBLE, NEUTRAL or IMPLAUSIBLE) and scored on a scale from 1.0 to 5.0. BLE, NEUTRAL, PLAUSIBLE) given the clarification2 and its context. • SubTask-B: Ranking. The goal is to predict the plausibility score on a scale from 1 to 5 given the clarification and its context. In this paper, we describe our winning system for both subtasks. We built our system based on a replaced token detection (RTD) task pre-training model. The idea is that the replaced token detection task is similar to this shared task which focuses on distinguishing semantically similar words/phrases. To close the gap the model trained between the pre- training phase …

Content truncated to 500 words