PLUJAGH at SemEval-2016 Task 11: Simple System for Complex Word Identification

Proceedings of SemEval-2016, pages 953–957, San Diego, California, June 16-17, 2016. c⃝2016 Association for Computational Linguistics PLUJAGH at SemEval-2016 Task 11: Simple System for Complex Word Identification Krzysztof Wróbel Jagiellonian University ul. Golebia 24 31-007 Krakow, Poland AGH University of Science and Technology al. Mickiewicza 30 30-059 Krakow, Poland kwrobel@agh.edu.pl Abstract This paper presents the description of a sys- tem which detects complex words. It solely uses information regarding the presence of a word in a prepared vocabulary list. The sys- tem outperforms multiple more advanced sys- tems and is ranked fourth for the shared task, with minimal loss to the best system. F-score optimization guaranteed the first place in this measurement. Different features are consid- ered and evaluated. Maximal bounds are pre- dicted. The rule “the simplest methods give the best results” is confirmed. 1 Introduction The goal of Complex Word Identification (CWI) is to detect words in a text that are complex (not easy to understand) for some group of people. CWI is one of the tasks of SemEval-2016 (Paetzold and Specia, 2016). CWI can be treated as the first step of Lexical Simplification (LS). LS was a task of SemEval-2012 (Specia et al., 2012). Complex words were identi- fied using n-grams, the length of the word, and the number of syllables (Ligozat et al., 2012; De Belder et al., 2010; Biran et al., 2011). The resources exploited in this task include Wikipedia, WordNet, Google Web 1T corpus (Sinha, 2012; Paetzold and Specia, 2015). Additional annotation of input sen- tences was performed by: a part-of-speech tagger, and word sense disambiguation (Amoia and Ro- manelli, 2012; Jauhar and Specia, 2012). A similar task is the prediction of the readability of a whole text. In comparison, in CWI, each word has to be scored. The applied methods are summa- rized in (D˛ebowski et al., 2015). This paper presents findings regarding the neces- sary data and the performed experiments. For the fi- nal submission, a simple system was chosen, which scored at fourth place. 2 Task Data Analysis It is important to notice the difference between train- ing and test data. Each sentence in the training set was annotated by 20 annotators. If at least one of them classified a word in a sentence as complex, it was marked as complex. The training data consists of 2237 classified words. On the other hand, each sentence in the test data (88221 classified words) was annotated by only one annotator. Complex words represent 31.56% of the words in the training data. Fortunately, organizers pub- lished the unaggregated annotations – every word in a sentence has 20 annotations. In this scenario, only 4.55% instances are classified as complex. A priori probability of the word being complex is important knowledge for the classification task. What is more, the organizers shared the baseline results for test data (Table 1). It shows that complex words represent 4.7% of instances in the test data – similar to training. 3 Resources and Methods Knowledge bases are essential to this …

Content truncated to 500 words