SemEval 2016 Task 11: Complex Word Identification

Proceedings of SemEval-2016, pages 560–569, San Diego, California, June 16-17, 2016. c⃝2016 Association for Computational Linguistics SemEval 2016 Task 11: Complex Word Identification Gustavo Henrique Paetzold and Lucia Specia Department of Computer Science University of Sheffield, UK {ghpaetzold1,l.specia}@sheffield.ac.uk Abstract We report the findings of the Complex Word Identification task of SemEval 2016. To cre- ate a dataset, we conduct a user study with 400 non-native English speakers, and find that complex words tend to be rarer, less ambigu- ous and shorter. A total of 42 systems were submitted from 21 distinct teams, and nine baselines were provided. The results high- light the effectiveness of Decision Trees and Ensemble methods for the task, but ultimately reveal that word frequencies remain the most reliable predictor of word complexity. 1 Introduction Complex Word Identification (CWI) is the task of deciding which words should be simplified in a given text. It is commonly connected with the task of Lexical Simplification (LS), which has as goal to replace complex words and expressions with sim- pler alternatives. In the usual LS pipeline, which was first introduced by (Shardlow, 2014), CWI is the first step. An effective CWI strategy can pre- vent LS approaches from replacing simple words, and hence prevent them from making grammatical and/or semantic errors. Early LS approaches (De- vlin and Tait, 1998; Carroll et al., 1999) do not in- clude CWI. As shown in (Paetzold and Specia, 2013; Shardlow, 2014), ignoring this step can considerably decrease the quality of the output produced by a sim- plifier. CWI has been gaining popularity in recent re- search. The LS approach in (Horn et al., 2014) employs an implicit CWI strategy in which a target word is only deemed complex if the LS model can find a candidate substitution which is simpler. Their results, however, show that the approach is unable to find simplifications for one third of the complex words in the dataset. (Shardlow, 2013b) presents the CW corpus: the first dataset for CWI. Although a relevant contribution, this dataset contains only 731 instances extracted automatically from the Sim- ple English Wikipedia edits, which raises concerns about its reliability and applicability. The results obtained by Shardlow (2013a) high- light some of the issues of the dataset. They use the CW corpus to compare the performance of three so- lutions to CWI: a Threshold-Based approach, a Sup- port Vector Machine (SVM), and a “Simplify Ev- erything” approach. In their experiments, the “Sim- plify Everything” approach achieves higher Accu- racy, Recall and F-scores than all other systems, sug- gesting that simplifying all words in a sentence is the most effective approach for CWI. These results are clearly counter intuitive and conflicting with the conclusions drawn in (Paetzold and Specia, 2013; Paetzold, 2013; Shardlow, 2014). In this paper we describe the first edition of the Complex Word Identification task, organized at Se- mEval 2016. This is an initiative that aims to pro- vide reliable resources and new insights for CWI, as well as to establish the state of …

Content truncated to 500 words