JUST-BLUE at SemEval-2021 Task 1: Predicting Lexical Complexity using BERT and RoBERTa Pre-trained Language Models

Proceedings of the 15th International Workshop on Semantic Evaluation (SemEval-2021), pages 661–666 Bangkok, Thailand (online), August 5–6, 2021. ©2021 Association for Computational Linguistics 661 JUST-BLUE at SemEval-2021 Task 1: Predicting Lexical Complexity using BERT and RoBERTa Pre-trained Language Models Tuqa Bani Yaseen Qusai Ismail Sarah Al-Omari Eslam Al-Sobh Malak Abdullah Jordan University of Science and Technology Irbid Jordan tmbaniyaseen19, qfismail20, ssalomari18, esalsobh19 @ cit.just.edu.jo mabdullah@just.edu.jo Abstract Predicting the complexity level of a word or a phrase is considered a challenging task. It is even recognized as a crucial step in numer- ous NLP applications, such as text rearrange- ments and text simplification. Early research treated the task as a binary classification task, where the systems anticipated the existence of a word’s complexity (complex versus uncom- plicated). Other studies had been designed to assess the level of word complexity using regression models or multi-labeling classifica- tion models. Deep learning models show a significant improvement over machine learn- ing models with the rise of transfer learning and pre-trained language models. This paper presents our approach that won the first rank in the SemEval-task1 (sub stask1). We have calculated the degree of word complexity from 0-1 within a text. We have been ranked first place in the competition using the pre-trained language models BERT and RoBERTa, with a Pearson correlation score of 0.788. keywords: Neuro-linguistic programming (NLP), Lexical Complexity Prediction(LCP), Deep Learning, RoBERTa, BERT. 1 Introduction Lexical complexity plays a significant role in the readability level and comprehension. The precise anticipation of lexical complexity can help systems direct the user to an acceptable simple text accu- rately or modify the text to be more fluid (Broth- ers and Traxler, 2016). Predicting the complexity of words is a subjective and challenging problem, while it is conjectural, too. Yet, mapping words into their complexity is an essential task to under- stand natural language. Numerous components can influence the prediction of lexical complexity. Sev- eral approaches were proposed to solve or mitigate this type of study using Machine and Deep learn- ing methods (Sengupta et al., 2020; Gooding and Kochmar, 2019; Bahja, 2020). This paper describes the JUST-BLUE team’s model that participated in the SemEval 2021-task1, Lexical Complexity Prediction (LCP) (Shardlow et al., 2021). The task provides participants with an augmented version of CompLex, a multi-domain English dataset with sentences annotated using a 5-point Likert scale (1-5) (from very easy to very difficult) (Shardlow et al., 2020). The task is to predict the complexity value of words in con- text. It is worth mentioning that our model, JUST- BLUE, has been ranked first in this task. We have used the pre-trained language models, BERT and RoBERTa Which have proven their effectiveness in this area (Liu et al., 2019), along with the ensem- bling method (weighted averaging) to achieve the highest Pearson correlation score of 0.788. The rest of this paper is organized as follows: Section 2 sheds light on related work. Section 3 de- scribes the methodology proposed in this research. Section 4 discusses the experimentation …

Content truncated to 500 words