Garuda & Bhasha at SemEval-2016 Task 11: Complex Word Identification Using Aggregated Learning Models

Proceedings of SemEval-2016, pages 1006–1010, San Diego, California, June 16-17, 2016. c⃝2016 Association for Computational Linguistics Garuda & Bhasha at SemEval-2016 Task 11: Complex Word Identification Using Aggregated Learning Models Prafulla Kumar Choubey and Shubham Pateria System Software Samsung R & D Institute India Bangalore Pvt Ltd Bangalore, Karanataka, India -560037 prafulla.ch, s.pateria@samsung.com Abstract This paper describes aggregated learning models for Complex Word Identification (CWI) task in SemEval 2016. The work focused on selecting the features that de- termine complexity of words and used dif- ferent combinations of support vector ma- chine (SVM) and decision tree (DT) tech- niques for classification. These classifiers were pipelined with pre-processing and post- processing blocks which helped improving ac- curacy of systems, though had little impact on recall. Four systems were evaluated on the test set; SVM and DT systems by team Bhasha achieved G score of 0.529 and 0.508 respec- tively and SVM&DT and SVMPP systems by team Garuda achieved G scores of 0.360 and 0.546 respectively. 1 Introduction CWI constitutes the first stage in lexical simplifica- tion (LS) pipeline (Specia et al., 2012). Performance of any LS system highly relies on accurate identifi- cation of complex words. (Shardlow, 2014) catego- rized errors in LS into six types and two of them (type 2A and 2B) are reflections of recall and preci- sion of CWI system. The author concluded that type 2B errors occur with highest frequency, nearly 0.54, followed by type 2A (0.11). Also only 0.1 of sim- plifications were free from error. Thus, designing a robust CWI system impacts LS task the most. CWI has long been researched and used in LS systems. (Paetzold, 2015) named some of the ap- proaches used inclusive to LS task, like lexicon based approaches, frequency thresholding based ap- proaches, word length thresholding and user driven approaches. Trained classifiers have also been pro- posed in literature like, (Shardlow, 2013) used SVM as classifier on feature sets comprising of frequency, length, synonym counts etc. This work is extension to the method proposed by (Shardlow, 2013). The proposed systems are built on broader set of features and multiple classifiers have been used individually or in aggregation to know trade-off between precision and recall. These sys- tems are trained solely on the training data released by task organizers and no external resource has been used, except the WordNet (Miller, 1995; Fellbaum, 1998). The rest of paper is organized as follows, sec- tion 2 describes the feature sets used. Section 3 de- scribes the approaches used and submitted systems performance w.r.t other participants. Finally, section 4 presents the conclusions and outlines some direc- tions of future work. 2 Feature Sets Selection of relevant feature is key to better perfor- mance of a machine learning model. While in some tasks, feature extraction is an obvious process, in tasks like CWI its hard to determine features apt- ness. So, in this work all the feature sets whose in- clusion improves any of recall, precision or accuracy were used. These features have been …

Content truncated to 500 words