RG PA at SemEval-2021 Task 1: A Contextual Attention-based Model with RoBERTa for Lexical Complexity Prediction

Proceedings of the 15th International Workshop on Semantic Evaluation (SemEval-2021), pages 623–626 Bangkok, Thailand (online), August 5–6, 2021. ©2021 Association for Computational Linguistics 623 RG PA at SemEval-2021 Task 1: A Contextual Attention-based Model with RoBERTa for Lexical Complexity Prediction Gang Rao∗, Maochang Li∗, Xiaolong Hou, Lianxin Jiang, Yang Mo, Jianping Shen Ping An Life Insurance, Lt ShenZhen, China {raogang532, limaochang389, houxiaolong430, jianglianxin769, moyang853, shenjianping324}@pingan.com.cn Abstract In this paper we propose a contextual attention- based model with two-stage fine-tune training using RoBERTa. First, we perform the first- stage fine-tune on corpus with RoBERTa, so that the model can learn some prior domain knowledge. Then we get the contextual em- bedding of context words based on the token- level embedding with the fine-tuned model. And we use Kfold cross-validation to get K models and ensemble them to get the final re- sult. Finally, we attain the 2nd place in the final evaluation phase of sub-task 2 with pear- son correlation of 0.8575. 1 Introduction LCP is an augmented version of Complex Word Identification(CWI) (Shardlow et al., 2020), pre- dict complexity score for each target word in a sentence. The dataset is a multi-domain English dataset annotated with a 5-point Likert scale (1-5). The annotation model in CompLex addresses com- plexity as a continuum instead of a binary feature. Previous studies of CWI treat the task as a binary classification, predict a complexity label (complex vs. non-complex) for a set of target words in a sentence. In this paper, we exploratory data analysis(EDA) for the dataset, and found that the distribution of the single task and multi task dataset are very in- conformity, so should bulid two models for every dataset great than one model for merge two task dataset. Several key technologies as follows: • Train a RoBERTa based fine-tune corpus clas- sifier. It use the data of all single and multi ∗These authors contributed equally to this work This work is licensed under a Creative Commons Attribution 4.0 In- ternational License. License details: http:// creativecom- mons.org/licenses/by/4.0/ with train, trial and test dataset as train data, no dev and test, and only train 1 epoch, which enable the RoBERTa model ahead of time learning the domain knowledge. • At each layer, calculate target vector and con- text tokens embedding attention, the layer con- text vector is average the context tokens em- bedding with soft alignment. • Weighted the RoBERTa last 12 layers context vector and target vector. They use the same weights, and it’s sum equals to 1. • The degeneration of gradual unfreezing (Howard and Ruder, 2018). At first epoch freeze the pretrained model parameter only learning the head layers parameter, then un- freeze all model parameter. • Multi-Sample Dropout at last layer (Inoue, 2019) 2 Background Previous approaches to CWI typically refer to bi- nary identification of complex words, two shared tasks on CWI topic have been organized so far. SemEval-2016 Task 11 (Paetzold and Specia, 2016) and BEA workshop 2018 (Yimam et al., 2018). The two tasks approache …

Content truncated to 500 words