CoastalCPH at SemEval-2016 Task 11: The importance of designing your Neural Networks right

Proceedings of SemEval-2016, pages 1028–1033, San Diego, California, June 16-17, 2016. c⃝2016 Association for Computational Linguistics CoastalCPH at SemEval-2016 Task 11: The importance of designing your Neural Networks right Joachim Bingel and Natalie Schluter Centre for Language Technology University of Copenhagen (Denmark) {bingel,natschluter}@hum.ku.dk H´ector Mart´ınez Alonso Alpage INRIA & University Paris 7 (France) hector.martinez-alonso@inria.fr Abstract We present two methods for the automatic de- tection of complex words in context as per- ceived by non-native English readers, for the SemEval 2016 Task 11 on Complex Word Identification (Paetzold and Specia, 2016). The submitted systems exploit the same set of features, but are highly disparate in (i) their learning algorithm and (ii) their angle on the learning objective, where especially the lat- ter presents an effort to account for the spar- sity of positive instances in the data as well as the large disparity between the distributions of positive instances in the training and test data. We further present valuable insights that we gained during intensive and extensive post- task experiments. Those revealed that despite poor results in the task, our neural network ap- proach is competitive with the systems achiev- ing the best results. The central contribution of this paper is therefore a demonstration of the aptitude of deep neural networks for the task of identifying complex words. 1 Introduction The identification of complex words plays an impor- tant role in the development of simplified reading resources. In particular, accurate automatic complex word identification strongly benefits lexical simplifi- cation (LS) as a first step in an LS pipeline (Paetzold and Specia, 2013; Shardlow, 2014). As such, accu- rate complex word identification may also be critical to higher-level tasks in text simplification, e.g. sen- tence compression, where the overall readability of a text is at least partly influenced by the difficulty of its vocabulary. However, the perceived difficulty of words does not generalise across different groups of readers. That is, a system trained on manual annotations of word difficulty among second-language learn- ers does not necessarily perform well in predicting words that are particularly difficult to persons of re- duced literacy. The 2016 SemEval Task 11 puts its focus on the identification of complex words for second-language learners of English. We chose to explore supervised methods for the task, employing a simple feed-forward neural net- work as well as a logistic regression classifier for two separate system submissions. Both systems take as input a single list of word and context features that we deemed to potentially indicate the complexity of words themselves and with respect to their context in the sentence. In this system description paper, we focus specif- ically on our neural network model. As noted in the task description paper (Paetzold and Specia, 2016), systems based on neural networks generally per- formed rather weakly in the task, which the organ- isers speculate to be a consequence of the small amount of training data. However, we show in post- hoc experiments how careful network design may lead to performance figures close …

Content truncated to 500 words