UNBNLP at SemEval-2021 Task 1: Predicting lexical complexity with masked language models and character-level encoders

Proceedings of the 15th International Workshop on Semantic Evaluation (SemEval-2021), pages 650–654 Bangkok, Thailand (online), August 5–6, 2021. ©2021 Association for Computational Linguistics 650 UNBNLP at SemEval-2021 Task 1: Predicting lexical complexity with masked language models and character-level encoders Milton King and Ali Hakimi Parizi and Samin Fakharian and Paul Cook Faculty of Computer Science, University of New Brunswick Fredericton, NB E3B 5A3, Canada {milton.king,ahakimi,samin.fakharian,paul.cook}@unb.ca Abstract In this paper, we present three supervised sys- tems for English lexical complexity predic- tion of single and multiword expressions for SemEval-2021 Task 1. We explore the use of statistical baseline features, masked language models, and character-level encoders to pre- dict the complexity of a target token in con- text. Our best system combines information from these three sources. The results indicate that information from masked language mod- els and character-level encoders can be com- bined to improve lexical complexity predic- tion. 1 Introduction SemEval 2021 Task 1 (Shardlow et al., 2021) fo- cuses on predicting the complexity of a target token in an English sentence on a scale from 0 (not very complex) to 1 (very complex). For example, the token land in the example below is judged to have a low complexity of 0.19. 1. Our land will yield its increase. On the other hand, the token doxycycline in the following example is judged to have a relatively higher complexity of 0.75.1 2. The reason these two lines were unresponsive to doxycycline is unknown. The dataset for this task was originally proposed in Shardlow et al. (2020), and includes sentences from three sources: a translated bible, European Parlia- ment proceedings, and a biomedical corpus. This shared task contains two sub-tasks with the first focusing on lexical complexity for single words, 1These example sentences and complexity scores are taken from the dataset provided for the shared task. which will be referred to as SINGLE, and the sec- ond focusing on complexity of multiword expres- sions, which will be referred to as MULTI. In this paper, we explore the use of statistical baseline features, masked language models, and character-level encoders to predict the complexity of a target token in context. We first consider these approaches individually and then consider super- vised methods for combining them. We evaluate our models with Pearson correlation (R), Spear- man correlation (Rho), mean absolute error (MAE), mean square error (MSE), and R-squared (R2). We apply all our models to both sub-tasks and find that we achieve our best results with a model that com- bines all three sources of information — baseline features, masked language modeling, and character- level encoding. Specifically, we achieve our best results with respect to Pearson correlation — the evaluation measure used for the official shared task system ranking — using a model that combines complexity predictions from two approaches, one based on a character-level encoder, and the other based on a masked language model, which further incorporates the baseline features, using support vector regression. Although we achieve our best results using this approach on …

Content truncated to 500 words