cs60075_team2 at SemEval-2021 Task 1 : Lexical Complexity Prediction using Transformer-based Language Models pre-trained on various text corpora

Proceedings of the 15th International Workshop on Semantic Evaluation (SemEval-2021), pages 678–682 Bangkok, Thailand (online), August 5–6, 2021. ©2021 Association for Computational Linguistics 678 cs60075 team2 at SemEval-2021 Task 1 : Lexical Complexity Prediction using Transformer-based Language Models pre-trained on various text corpora Abhilash Nandy Sayantan Adak Tanurima Halder Sai Mahesh Pokala {nandyabhilash, sayantanadak.skni, haldertanurima, pokalasaimahesh}@gmail.com Indian Institute of Technology, Kharagpur, India Abstract This paper describes the performance of the team cs60075 team2 at SemEval 2021 Task 1 - Lexical Complexity Prediction. The main contribution of this paper is to fine- tune transformer-based language models pre- trained on several text corpora, some being general (E.g., Wikipedia, BooksCorpus), some being the corpora from which the CompLex Dataset was extracted, and others being from other specific domains such as Finance, Law, etc. We perform ablation studies on select- ing the transformer models and how their indi- vidual complexity scores are aggregated to get the resulting complexity scores. Our method1 achieves a best Pearson Correlation of 0.784 in sub-task 1 (single word) and 0.836 in sub-task 2 (multiple word expressions). 1 Introduction Complex words hinder the readability of a text, as discussed in (William, 2004). To mitigate this problem, there is a necessity of lexical simplifica- tion (Leroy et al., 2013), and predicting the com- plexity of words is an integral part of this process. Language Models learn the probability of co- occurrence of words in a corpus. They have been used for various sentence completion and text-based classification tasks. The first language models were n-gram Markov Models (Rabiner and Juang, 1986), which performed well for tasks that did not require very long-range dependen- cies. Then came RNNs (Cho et al., 2014a), LSTMs (Hochreiter and Schmidhuber, 1997) and GRUs (Cho et al., 2014b), which were able to un- derstand longer contexts, but struggled with long paragraphs due to the vanishing gradient prob- lem. Transformers (Vaswani et al., 2017) were a task-agnostic solution that performed better due 1The code is available at https://github.com/ abhi1nandy2/CS60075-Team-2-Task-1 to the presence of Attention Layers between hid- den layers of the neural network, which helped the layers of the neural network to look at the en- tire input at once. Transformers can perform very well on a broad suite of tasks by fine-tuning on a small number of task-specific samples. The in- tuition behind using such transformer-based lan- guage models for Lexical Complexity Prediction (LCP) was - transformer models pre-trained on different corpora would mimic annotators (of the CompLex Dataset (com)) having different back- grounds. Since the final score is an aggregation of the annotation scores given by annotators, we ag- gregate the various scores that are given as outputs by the transformer-based models fine-tuned on the CompLex Dataset. The rest of the paper is organized as follows. Section 2 gives an overview of our solution ap- proach, Section 3 talks about the corpora used for pre-training and the dataset used for fine-tuning for the LCP task, Section 4 discusses the experi- mental settings, baselines used, and a comparison …

Content truncated to 500 words