Sensible at SemEval-2016 Task 11: Neural Nonsense Mangled in Ensemble Mess

Proceedings of SemEval-2016, pages 963–968, San Diego, California, June 16-17, 2016. c⃝2016 Association for Computational Linguistics Sensible at SemEval-2016 Task 11: Neural Nonsense Mangled in Ensemble Mess Nat Gillin Gillin Inc. 36 Natchez Street, Seahaven, USA gillin.nat@gmail.com Abstract This paper describes our submission to the Complex Word Identification (CWI) task in SemEval-2016. We test an experimental ap- proach to blindly use neural nets to solve the CWI task that we know little/nothing about. By structuring the input as a series of se- quences and the output as a binary that indi- cates 1 to denote complex words and 0 other- wise, we introduce a novel approach to com- plex word identification using Recurrent Neu- ral Nets (RNN). We also show that it is pos- sible to simply ensemble several RNN clas- sifiers when we are unsure of the optimal hyper-parameters or the best performing mod- els using eXtreme gradient boosted trees clas- sifiers. Our systems submitted to the CWI task achieved the highest accuracy and F-score among the systems that uses neural networks. 1 Introduction The Deep Learning Tsunami has hit the Natu- ral Language Processing (NLP) and Computational Linguistics field (Manning, 2016). Deep neural nets has shown to be the ultimate hammer in various NLP shared tasks, systems trained on neural nets often emerge as the top systems and/or beat state-of-the- art performance (Collobert et al., 2011; Mikolov et al., 2013; Pennington et al., 2014; Levy et al., 2014; Shazeer et al., 2016; Gupta et al., 2015; Jean et al., 2015; Kreutzer et al., 2015; Sultan et al., 2014; Sul- tan et al., 2015) In the concluding remarks of the Google’s Deep Learning course on Udacity1, Vincent Vanhoucke 1https://www.udacity.com/course/deep-learning–ud730 said, “What’s really cool about those [neural net ap- plication] examples is that you don’t have to know much about the problem you’re trying to solve”. Armed with basic knowledge of deep learning and neural nets and almost zero familiarity of the prob- lem, we attempt to treat the Complex Word Iden- tification (CWI) task as a binary classification task using Long Short-Term Memory (LSTM) Recur- rent Neural Nets (RNN) with Gated Recurrent Units (GRU). 2 Neural Network and Deep Learning Neural Networks are powerful at modelling various modalities, e.g. signals, text, images, videos. As the name suggests, neural network is inspired by the brain’s synaptic transmission mechanism that trans- mits signalling molecules (aka neurotransmitters) to different signal receptors (aka neurons) throughout our body. Metaphorically, we can emulate a neu- ron as a computational unit and consider the neuro- transmitters as real number inputs and outputs that pass from a neuron to another. Each input to a neu- ron comes with a associated weight and the neuron will process the different inputs (often by summing them) and passing it to a non-linear function which will provide an output value. For instance, we can think of a neuron as a typi- cal AND/OR logic gate. Given two binary inputs x1 and x2 and a bias unit with input 1 and a …

Content truncated to 500 words