Melbourne at SemEval 2016 Task 11: Classifying Type-level Word Complexity using Random Forests with Corpus and Word List Features

Proceedings of SemEval-2016, pages 975–981, San Diego, California, June 16-17, 2016. c⃝2016 Association for Computational Linguistics Melbourne at SemEval 2016 Task 11: Classifying Type-level Word Complexity using Random Forests with Corpus and Word List Features Julian Brooke Timothy Baldwin Computing and Info Systems The University of Melbourne jabrooke@unimelb.edu.au tb@ldwin.net Alexandra L. Uitdenbogerd Computer Science and Info Technology RMIT University sandrau@rmit.edu.au Abstract SemEval 2016 task 11 involved determining whether words in a sentence were complex or simple for a cohort of people with English as a second language. Training data consisted of 200 annotated sentences, representing the combined judgements of 20 human annota- tors, such that if any annotator of the group labelled a word as complex, then it was con- sidered to be complex. Testing was based on single annotator judgements. Our system used a random forest classifier with a variety of fea- tures, the most important of which were term frequency statistics garnered from four large corpora, and style lexicons built on two large corpora. Minor features in the final system include the presence or absence of words in various readability word lists; many other fea- tures we tried were not successful. Our rank- ing amongst submitted systems did not reflect the strength of our system, due to submitting a far from optimal weighting between complex and simple, but we show that when a more ap- propriate weighting is used, our system ranks amongst the best submitted systems. 1 Introduction Most work related to readability measurement (Chall and Dale, 1995) focuses on text-level assess- ment, but it is clear that being able to determine the difficulty of individual words is important to both that task as well as related ones such as lexi- cal text simplification (Shardlow, 2014). Although some words can be considered conceptually difficult — that is, a level of intellectual sophistication is re- quired to grasp its meaning — for language learners, it is more common for words to be considered dif- ficult (or complex) simply because a reader has had little or no exposure to them. This exposure may depend on many different external factors related to the person’s background, some of which may gener- alize across other similar readers, while others may be entirely idiosyncratic to the reader in question. For example, those who study academic English, or operate in an academic environment, have a different vocabulary exposure to those who specialize in hos- pitality English. Therefore, there is value in not only trying to predict some prior difficulty of a word, but also trying to generalize across readers in a similar cohort. Task 11 of the 2016 SemEval competition (Complex Word Identification) is aimed at address- ing this challenge. This paper describes our system for the task. We commenced with previous work in word readability scoring (Brooke et al., 2012) and stylistic lexicon creation (Brooke and Hirst, 2013; Brooke and Hirst, 2014). For features, we drew on a diverse set of corpus-based and human-derived metrics, and built a random forest-based classifier. While a …

Content truncated to 500 words