Saagie at Semeval-2019 Task 5: From Universal Text Embeddings and Classical Features to Domain-specific Text Classification

Proceedings of the 13th International Workshop on Semantic Evaluation (SemEval-2019), pages 469–475 Minneapolis, Minnesota, USA, June 6–7, 2019. ©2019 Association for Computational Linguistics 469 Saagie at Semeval-2019 Task 5: From Universal Text Embeddings and Classical Features to Domain-specific Text Classification Miriam Benballa Saagie / LITIS Rouen, France miriam.benballa@saagie.com Sebastien Collet Saagie Rouen, France sebastien.collet@saagie.com Romain Picot-Clemente Saagie Rouen, France romain.picotclemente@saagie.com Abstract This paper describes our contribution to Se- mEval 2019 Task 5: Hateval. We propose to investigate how domain-specific text classifi- cation task can benefit from pretrained state of the art language models and how they can be combined with classical handcrafted features. For this purpose, we propose an approach based on a feature-level Meta-Embedding to let the model choose which features to keep and how to use them. 1 Introduction In this paper, we describe our system for Task 5 of SemEval 2019 (Basile et al., 2019), namely Mul- tilingual detection of hate speech against immi- grants and women in Twitter (HatEval). In this task, participants were asked to automatically clas- sify English and Spanish tweets as hateful or not for Subtask A, and to predict if these tweets are aggressive or not, then identify whether the target is generic or individual for Subtask B. We partici- pated in all subtasks for both English and Spanish. Our main interest in this competition is to evalu- ate how a domain-specific dataset can take advan- tage of unsupervised data and moreover, how very different features can be combined efficiently in a deep neural network to improve classification. For this purpose, we propose to exploit state of the art pretrained deep learning models in text classifica- tion and classical features into an architecture that allows combining them dynamically. Our work consists of three steps: features cre- ation, dynamic meta-embedding and finally com- bining this information to classify tweets. The next sections are organized as follows: in section 2, we will briefly cover the related work, in sec- tion 3 we will explain our model, then in section 4 we will expose our experiences, and finally we will introduce our results in section 5. 2 Related Work A successful classical approach for tweets clas- sification and sentiment analysis is to use neural networks on top of pre-trained word embeddings. Word embeddings are trained with unsupervised data with a method called distant supervision (Go et al., 2009). Deriu et al. (2016) use Convolu- tional neural networks on top of those word em- beddings while Cliche (2017) is using an ensemble of CNNS and LSTMs. Both solutions won respec- tively SemEval task 4 in 2016 and 2017. For tasks more closely related to SemEval Task 5, S´anchez G´omez (2018) won the IberEval 2018 Aggressiveness detection task with an Ensembling of several SVMs models. The Ensembling is done with a Genetic Algorithm. Cuza et al. (2018) pro- pose a model with a Bi-LSTMs network with at- tention layers on top of pre-trained word embed- dings. Their solution got the 2nd place. On the Mysogyny detection task …

Content truncated to 500 words