SyntNN at SemEval-2018 Task 2: is Syntax Useful for Emoji Prediction? Embedding Syntactic Trees in Multi Layer Perceptrons

Proceedings of the 12th International Workshop on Semantic Evaluation (SemEval-2018), pages 477–481 New Orleans, Louisiana, June 5–6, 2018. ©2018 Association for Computational Linguistics SyntNN at SemEval-2018 Task 2: is Syntax Useful for Emoji Prediction? Embedding Syntactic Trees in Multi Layer Perceptrons Andrea Santilli Department of Enterprise Engineering University of Rome Tor Vergata Italy andrea.santilli@live.it Fabio Massimo Zanzotto Department of Enterprise Engineering University of Rome Tor Vergata Italy fabio.massimo.zanzotto@uniroma2.it Abstract In this paper, we present SyntNN as a way to include traditional syntactic mod- els in multilayer neural networks used in the task of Semeval Task 2 of emoji pre- diction (Barbieri et al., 2018). The model builds on the distributed tree embedder also known as distributed tree kernel (Zanzotto and Dell’Arciprete, 2012). Initial results are ex- tremely encouraging but additional analysis is needed to overcome the problem of overfitting. 1 Introduction Syntactic models of language have always played a crucial role in many natural language processing tasks but, in recent years, it has been marginalized by the advent of neural networks and in particular long-short term memory (LSTM). These latter net- works have had a tremendous impact on how lin- guistic tasks are modeled and, sometimes, solved. In this paper, we want to explore the use of “tra- ditional” syntactic information within a neural net- work framework in the task of Semeval Task 2 of emoji prediction (Barbieri et al., 2018, 2017). We propose SyntNN as a way to include traditional syntactic models in multilayer neural networks. The model builds on the distributed tree embed- der also known as distributed tree kernel (Zanzotto and Dell’Arciprete, 2012; Ferrone and Zanzotto, 2014; Zanzotto et al., 2015) that is a way to trans- pose syntactic information in small vectors. Initial results are extremely encouraging: SyntNN out- performs syntax-unaware neural networks on the trial set. Unfortunately, these promising results are not confirmed on the test set. Hence, we analyzed these results to try to understand why this has hap- pened. 2 SyntNN: a Syntax-aware Multilayer Perceptron SyntNN is a simple, non-recurrent neural network that aims to exploit traditional syntactic interpre- tations of tweets in classification tasks. This net- work wants to explore two questions: first, in- vestigating whether “traditional” syntactic inter- pretation can be used within neural networks and, second, understanding if syntactic information is useful in modeling tweets for the specific task of emoji prediction. The architecture of SyntNN is then organized around two sub-networks (see Fig. 1): (1) a syntactic sub-network and (2) a semantic sub- network. These two sub-networks are then merged to obtain the final classification layer. The rest of the section describes the two sub- networks and the merging layer. 2.1 Syntactic Subnetwork with a Distributed Tree Embedding Layer The syntactic subnetwork aims to take syntactic interpretations of tweets and make them available to a simple, non-recurrent neural network. The key point is how to obtain the transformation of the symbolic representation of syntactic trees into tensor-based representations that have meaningful properties. The Distributed Tree Embedding Layer (DTE) (see …

Content truncated to 500 words