LTG at SemEval-2016 Task 11: Complex Word Identification with Classifier Ensembles

Proceedings of SemEval-2016, pages 996–1000, San Diego, California, June 16-17, 2016. c⃝2016 Association for Computational Linguistics LTG at SemEval-2016 Task 11: Complex Word Identification with Classifier Ensembles Shervin Malmasi1 Mark Dras1 Marcos Zampieri2,3 1Macquarie University, Sydney, NSW, Australia 2Saarland University, Germany 3German Research Center for Artificial Intelligence, Germany {first.last}@mq.edu.au, marcos.zampieri@dfki.de Abstract We present the description of the LTG entry in the SemEval-2016 Complex Word Identi- fication (CWI) task, which aimed to develop systems for identifying complex words in En- glish sentences. Our entry focused on the use of contextual language model features and the application of ensemble classification meth- ods. Both of our systems achieved good per- formance, ranking in 2nd and 3rd place overall in terms of F-Score. 1 Introduction Complex Word Identification (CWI) is the task of identifying complex words in texts using computa- tional methods (Shardlow, 2013). The task is usu- ally carried out as part of lexical and text simpli- fication systems. Shardlow (2014) considers CWI as the first processing step in lexical simplification pipelines. Complex or difficult words should first be identified so they can be later substituted by simpler ones to improve text readability. CWI has gained more importance in the last decade as lexical and text simplification systems have been developed or tailored for a number of pur- poses. They have been applied to make texts more accessible to language learners (Petersen and Os- tendorf, 2007); other researchers have explored text simplification strategies targeted at populations with low literacy skills (Alu´ısio et al., 2008). Finally, an- other relevant application of text simplification are people with dyslexia (Rello et al., 2013). The SemEval 2016 Task 11: Complex Word Iden- tification (CWI) provides an interesting opportunity to evaluate methods and approaches for this task. The organizers proposed a binary text classification task in which participants were required label words in English sentences as either complex (1) or simple (0). The task organizers provided participants with a training set containing sentences annotated with this information, followed by an unlabeled test set for evaluation. The assessment of whether words in a sentence are complex or simple was performed by human annotators required to label the data.1 2 Data Based on the information available at the shared task’s website2: “400 annotators were presented with several sentences and asked to select which words they did not understand their meaning.” The CWI task dataset was divided as follows: • Training set: 2,237 judgments by 20 annota- tors over 200 sentences. A word is considered complex if at least one of the 20 annotators as- signed it as so. • Test set: 88,221 judgments made over 9,000 sentences (1 annotator per sentence). 1Here the term complex is used as a synonym for difficult. Unlike the Morphology term complex (antonym of simplex) that defines compound words or words composed of multiple morphs (Adams, 2001). 2http://alt.qcri.org/semeval2016/task11/ 996 3 Methodology The primary focus of our team’s entry was the use of judgements from different annotators to create train- ing data. We looked at how …

Content truncated to 500 words