BigGreen at SemEval-2021 Task 1: Lexical Complexity Prediction with Assembly Models

Proceedings of the 15th International Workshop on Semantic Evaluation (SemEval-2021), pages 667–677 Bangkok, Thailand (online), August 5–6, 2021. ©2021 Association for Computational Linguistics 667 BigGreen at SemEval-2021 Task 1: Lexical Complexity Prediction with Assembly Models Aadil Islam, Weicheng Ma, Soroush Vosoughi Department of Computer Science Dartmouth College {aadil.islam.21,weicheng.ma.gr,soroush.vosoughi}@dartmouth.edu Abstract This paper describes a system submitted by team BigGreen to LCP 2021 for predict- ing the lexical complexity of English words in a given context. We assemble a feature engineering-based model with a deep neu- ral network model founded on BERT. While BERT itself performs competitively, our fea- ture engineering-based model helps in extreme cases, eg. separating instances of easy and neutral difficulty. Our handcrafted features comprise a breadth of lexical, semantic, syn- tactic, and novel phonological measures. Vi- sualizations of BERT attention maps offer in- sight into potential features that Transformers models may learn when fine-tuned for lexical complexity prediction. Our ensembled predic- tions score reasonably well for the single word subtask, and we demonstrate how they can be harnessed to perform well on the multi word expression subtask too. 1 Introduction Lexical simplification (LS) is the task of replacing difficult words in text with simpler alternatives. It is relevant in reading comprehension, where early studies have shown infrequent words lead to more time spent by a reader fixated on it, and that ambi- guity in a word’s meaning adds to comprehension time (Rayner and Duffy, 1986). Complex word identification (CWI) is believed to be a fundamen- tal step in the automation of lexical simplification (Shardlow, 2014). Early techniques for conducting CWI suffer from a lack of robustness, from simpli- fying all words to then study its efficacy (Devlin, 1998), to applying thresholds on features like word frequency (Zeng et al., 2005). This year’s Lexical Complexity Prediction (LCP) shared task (Shardlow et al., 2021) forgoes the treat- ment of word difficulty as a binary classification task (Paetzold and Specia, 2016a; Yimam et al., 2018) and instead measures degree of complexity on a continuous scale. This choice is intriguing as it mitigates a dilemma with previous approaches of having to treat words extremely close to a deci- sion boundary (suppose a threshold deems a word’s difficulty) identically to those that are far away, ie. extremely easy or extremely difficult. Teams are asked to submit predictions on un- labeled test sets for two subtasks: predicting on English single word and multi word expressions (MWEs). For each subtask, BigGreen presents a machine learning-based approach that fuses the pre- dictions of a feature engineering-based regressor with those of a feature learning-based deep neural network model founded on BERT (Devlin et al., 2018). Our code is made available on GitHub.1 2 Related Work Previous studies have looked at estimating the readability of a given text at the sentence-level. Mc Laughlin (1969) regresses the number of poly- syllabic words in a given lesson against the mean score for students quizzed on said lesson, yielding the SMOG Readability Formula. Dale and Chall (1948) offer a list of …

Content truncated to 500 words