PALI at SemEval-2022 Task 7: Identifying Plausible Clarifications of Implicit and Underspecified Phrases in Instructional Texts

Proceedings of the 16th International Workshop on Semantic Evaluation (SemEval-2022), pages 1084 - 1089 July 14-15, 2022 ©2022 Association for Computational Linguistics PALI at SemEval-2022 Task 7: Identifying Plausible Clarifications of Implicit and Underspecified Phrases in Instructional Texts Mengyuan Zhou and Dou Hu and Mengfei Yuan and Meizhi Jin and Xiyang Du and Lianxin Jiang and Yang Mo and Xiaofeng Shi Ping An Life Insurance Company of China, Ltd. {ZHOUMENGYUAN425, HUDOU470, YUANMENGFEI854, JINMEIZHI005, DUXIYANG037, JIANGLIANXIN769, MOYANG853, SHIXIAOFENG309} @pingan.com.cn Abstract This paper describes our system used in the SemEval-2022 Task 7(Roth et al.): Identifying Plausible Clarifications of Implicit and Under- specified Phrases. Semeval Task7 is an more complex cloze task, different than normal cloze task, only requiring NLP system could find the best fillers for sentence. In Semeval Task7, NLP system not only need to choose the best fillers for each input instance, but also evaluate the quality of all possible fillers and give them a relative score according to context semantic information. We propose an ensemble of differ- ent state-of-the-art transformer-based language models(i.e., RoBERTa and Deberta) with some plug-and-play tricks, such as Grouped Layer- wise Learning Rate Decay (GLLRD) strategy, contrastive learning loss, different pooling head and an external input data preprecess block be- fore the information came into pretrained lan- guage models, which improve performance sig- nificantly. The main contributions of our sys- tem are 1) revealing the performance discrep- ancy of different transformer-based pretraining models on the downstream task; 2) presenting an efficient learning-rate and parameter atten- uation strategy when fintuning pretrained lan- guage models; 3) adding different constrative learning loss to improve model performance; 4) showing the useful of the different pooling head structure. Our system achieves a test ac- curacy of 0.654 on subtask1(ranking 4th on the leaderboard) and a test Spearman’s rank corre- lation coefficient of 0.785 on subtask2(ranking 2nd on the leaderboard). 1 Introduction Cloze tasks have become a standard framework for evaluating various discourse-level phenomena in NLP. Some prominent examples include the narra- tive cloze test(Hoshino and Nakagawa, 2007), the story cloze test (Xie et al., 2020), and the LAM- BADA word prediction task(Paperno et al., 2016). In these tasks, NLP systems are required to make a prediction about the filler of a cloze that is most likely to continue the discourse. However, these ex- isting cloze tasks focus on the accuray of choosen fillers, ignore evaluating the absolute quality of all possible predictions. The goal of Semeval 2022 Task7 is to evaluate the ability of NLP systems to distinguish between plausible and implausible clarifications of an in- struction. The task is formulated as a complex cloze task, which involve two sub tasks. In Sub task 1, for each(sentence, filler) pair, NLP system need to classify four fillers into plausible, implausible or neutral and the evaluating indicator is accuracy. In Sub task 2, for each pair, NLP system need to predict scores for five fillers and the evaluating in- dicator is Spearman’s rank correlation coefficient. Since 2018, NLP models have adopted the con- …

Content truncated to 500 words