archer at SemEval-2021 Task 1: Contextualising Lexical Complexity

Proceedings of the 15th International Workshop on Semantic Evaluation (SemEval-2021), pages 694–699 Bangkok, Thailand (online), August 5–6, 2021. ©2021 Association for Computational Linguistics 694 archer at SemEval-2021 Task 1: Contextualising Lexical Complexity Irene Russo ILC-CNR, Pisa, Italy irene.russo@ilc.cnr.it Abstract Evaluating the complexity of a target word in a sentential context is the aim of the Lexical Complexity Prediction task at SemEval-2021. This paper presents the system created to as- sess single words lexical complexity, combin- ing linguistic and psycholinguistic variables in a set of experiments involving random forest and XGboost regressors. Beyond encoding out-of-context information about the lemma, we implemented features based on pre-trained language models to model the target word’s in-context complexity. 1 Introduction Lexical complexity prediction is the task aiming at evaluating the complexity of a word in context, modeling a crucial aspect of reading comprehen- sion. Complex words can slow down the read- ing process; there is a well-known correlation be- tween a word’s difficulty and the time spent looking at it, as emerging from eye-tracking experiments (Mousikou et al., 2021). Assessing the complexity of a specific word in context is a crucial prerequisite for NLP systems aiming to evaluate a text’s readability and produce a simplified version of it. It is a prerequisite for text simplification systems based on lexical sub- stitutions and can be the starting point to tailor a text to the user’s needs. It is a topic worthy of investigation from multiple points of view. In the past, datasets containing crowdsourced evaluations of lexical items’ complexity (Paetzold and Specia, 2016b; ˇStajner et al., 2018) have been used in eval- uation campaigns. In this paper, we introduce the system used to assess single English words lexical complexity at SemEval-2021 Lexical Complexity Prediction task (Shardlow et al., 2021). Based on previous ap- proaches to this issue, we combine linguistic and psycholinguistic variables, using a random forest regressor and an XGboost regressor. The major- ity of the variables encode information about the word out-of-context (e.g., frequency, number of letters, age of acquisition) without considering the sentential context and how it can affect an item’s complexity. To approximate in-context complexity, we con- sider the cloze probability of a word as its proba- bility to complete a particular sentence frame. We experiment with different language models in a masked word prediction framework, taking into ac- count the first ten most probable words occurring in that context. 2 Related works A wide range of approaches has been used for lexi- cal complexity prediction in past evaluation cam- paigns. However, previous tasks focused on classi- fication since the proposed datasets labeled words in context as easy or difficult. Including more classes makes the task more dif- ficult. (Gar´ı Soler et al., 2018) investigate the role of word embeddings in lexical complexity predic- tion for French words, using as training sets two French lexical resources that encode the distribu- tion of words across different levels of difficulty. According to the authors, the task is influenced by the context of use of …

Content truncated to 500 words