CS-UM6P at SemEval-2021 Task 1: A Deep Learning Model-based Pre-trained Transformer Encoder for Lexical Complexity

Proceedings of the 15th International Workshop on Semantic Evaluation (SemEval-2021), pages 585–589 Bangkok, Thailand (online), August 5–6, 2021. ©2021 Association for Computational Linguistics 585 CS-UM6P at SemEval-2021 Task 1: A Deep Learning Model-based Pre-trained Transformer Encoder for Lexical Complexity Nabil El Mamoun1 Abdelkader El Mahdaouy2 Abdellah El Mekki2 Kabil Essefar2 Ismail Berrada2 1Faculty of Sciences Dhar EL Mahraz, Sidi Mohamed Ben Abdellah University, Morocco 2School of Computer Sciences, Mohammed VI Polytechnic University, Morocco {firstname.lastname}@um6p.ma Abstract Lexical Complexity Prediction (LCP) involves assigning a difficulty score to a particular word or expression, in a text intended for a target audience. In this paper, we intro- duce a new deep learning-based system for this challenging task. The proposed system consists of a deep learning model, based on a pre-trained transformer encoder, for word and Multi-Word Expression (MWE) complex- ity prediction. First, on top of the encoder’s contextualized word embedding, our model employs an attention layer on the input context and the complex word or MWE. Then, the at- tention output is concatenated with the pooled output of the encoder and passed to a regres- sion module. We investigate both single-task and joint training on both Sub-Tasks data us- ing multiple pre-trained transformer-based en- coders. The obtained results are very promis- ing and show the effectiveness of fine-tuning pre-trained transformers for LCP tasks. 1 Introduction Text Simplification (TS) is a fundamental task for improving text readability, and presents a wide variety of use cases, including assisting children with reading difficulties and native speakers with low literacy levels (De Belder and Moens, 2010; Alu´ısio and Gasperin, 2010), increasing accessibil- ity for people with intellectual disabilities (Saggion, 2017), and facilitating certain aspects of language for language learners (Paetzold and Specia, 2016). TS may involve modifications to the syntactic struc- ture of a sentence, its lexical units or both (Shard- low, 2014). Lexical Simplification (LS), as a sub-task of TS, focuses on simplifying complex words of an in- put sentence. It first identifies complex words in a sentence, known as Complex Words Identifica- tion (CWI) or Lexical Complexity Prediction (LCP) task. Then, it replaces them with other alternatives of equivalent meaning. Those substitutions should be more simplistic while preserving the semantic and the grammatical structure of the input sentence (Paetzold and Specia, 2017; Qiang et al., 2020). Most of the previous research has modeled LCP as a binary classification task (Paetzold and Specia, 2017; Zampieri et al., 2016; Ronzano et al., 2016). A recent research study has introduced a multi- domain dataset, annotated using a 5-point Likert scale scheme (Shardlow et al., 2020). The aim is to label the complexity of a word or a Multi-Word Expression (MWE), in a more fine-grained manner, from very easy to very difficult. Hence, the lexical complexity of words is expressed on a continuous scale. In this paper, we introduce our submitted sys- tem to the SemEval-2021 LCP 1 and 2 Sub-Tasks (Shardlow et al., 2021). The proposed system con- sists of a deep learning model for word and MWE complexity …

Content truncated to 500 words