UTFPR at SemEval-2021 Task 1: Complexity Prediction by Combining BERT Vectors and Classic Features

Proceedings of the 15th International Workshop on Semantic Evaluation (SemEval-2021), pages 617–622 Bangkok, Thailand (online), August 5–6, 2021. ©2021 Association for Computational Linguistics 617 UTFPR at SemEval-2021 Task 1: Complexity Prediction by Combining BERT Vectors and Classic Features Gustavo Henrique Paetzold Universidade Tecnol´ogica Federal do Paran´a - Campus Toledo Toledo-PR, Brazil ghpaetzold@utfpr.edu.br Abstract We describe the UTFPR systems submitted to the Lexical Complexity Prediction shared task of SemEval 2021. They perform complexity prediction by combining classic features, such as word frequency, n-gram frequency, word length, and number of senses, with BERT vec- tors. We test numerous feature combinations and machine learning models in our experi- ments and find that BERT vectors, even if not optimized for the task at hand, are a great complement to classic features. We also find that employing the principle of compositional- ity can potentially help in phrase complexity prediction. Our systems place 45th out of 55 for single words and 29th out of 38 for phrases. 1 Introduction Accurately measuring the complexity of words can be useful in many ways. It facilitates the creation of text simplification technologies that could, for example, help in identifying and adapting challeng- ing excerpts of literary pieces targeting specific groups, such as children (De Belder and Moens, 2010) and second language learners (Paetzold and Specia, 2016e), and make news articles and official documents more accessible to the general popula- tion (Paetzold and Specia, 2016a). This task has received a considerable amount of attention in the past few years, especially due to the popularity of the Complex Word Identification (CWI) shared tasks of 2016 (Paetzold and Spe- cia, 2016c) and 2018 (Yimam et al., 2018), where dozens of teams were challenged to judge the com- plexity of words in context. While the CWI 2016 task used a simple binary complex/not complex classification setup for English only, the CWI 2018 task explored both a binary classification and a regression setup and multiple languages. The majority of the most successful systems sub- mitted to these shared tasks combined ensemble methods, such as Random Forests (Ho, 1995) and AdaBoost (Freund and Schapire, 1997) with numer- ous linguistic features, including word frequencies, n-gram frequencies, word length, number of senses, number of syllables, psycholinguistic metrics, and word embeddings (Konkol, 2016; Malmasi et al., 2016; Paetzold and Specia, 2016d; Gooding and Kochmar, 2018; Hartmann and Dos Santos, 2018). However, because these tasks were held prior to the ascension of transformer-based masked language models, such as BERT (Devlin et al., 2019) and ROBERTA (Liu et al., 2019), we could not find any systems that exploited the power of the features produced by them. In this paper, we describe the UTFPR systems for the Lexical Complexity Prediction shared task of SemEval 2021 (LCP 2021), which combine clas- sic complexity prediction features with contextual word and phrase representations extracted from transformer-based models. In our experiments, we explore the efficacy of a number of different ma- chine learning models, feature combinations, and corpora sources for our features. In what follows, …

Content truncated to 500 words