Deep Learning in Semantic Kernel Spaces

Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics, pages 345–354 Vancouver, Canada, July 30 - August 4, 2017. c⃝2017 Association for Computational Linguistics https://doi.org/10.18653/v1/P17-1032 Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics, pages 345–354 Vancouver, Canada, July 30 - August 4, 2017. c⃝2017 Association for Computational Linguistics https://doi.org/10.18653/v1/P17-1032 Deep Learning in Semantic Kernel Spaces Danilo Croce Simone Filice Giuseppe Castellucci Roberto Basili Department of Enterprise Engineering University of Roma Tor Vergata, Via del Politecnico 1, 00133, Rome, Italy {croce,filice,basili}@info.uniroma2.it castellucci@ing.uniroma2.it Abstract Kernel methods enable the direct usage of structured representations of textual data during language learning and inference tasks. Expressive kernels, such as Tree Kernels, achieve excellent performance in NLP. On the other side, deep neural net- works have been demonstrated effective in automatically learning feature representa- tions during training. However, their in- put is tensor data, i.e., they cannot man- age rich structured information. In this pa- per, we show that expressive kernels and deep neural networks can be combined in a common framework in order to (i) ex- plicitly model structured information and (ii) learn non-linear decision functions. We show that the input layer of a deep ar- chitecture can be pre-trained through the application of the Nystr¨om low-rank ap- proximation of kernel spaces. The result- ing “kernelized” neural network achieves state-of-the-art accuracy in three different tasks. 1 Introduction Learning for Natural Language Processing (NLP) requires to more or less explicitly account for trees or graphs to express syntactic and semantic in- formation. A straightforward modeling of such information has been obtained in statistical lan- guage learning with Tree Kernels (TKs) (Collins and Duffy, 2001), or by means of structured neu- ral models (Hochreiter and Schmidhuber, 1997; Socher et al., 2013). In particular, kernel-based methods (Shawe-Taylor and Cristianini, 2004) have been largely applied in language processing for alleviating the need of complex activities of manual feature engineering (e.g., (Moschitti et al., 2008)). Although ad-hoc features are adopted by many successful approaches to language learning (e.g., (Gildea and Jurafsky, 2002)), kernels pro- vide a natural way to capture textual generaliza- tions directly operating over (possibly complex) linguistic structures. Sequence (Cancedda et al., 2003) or tree kernels (Collins and Duffy, 2001) are of particular interest as the feature space they im- plicitly generate reflects linguistic patterns. On the other hand, Recursive Neural Networks (Socher et al., 2013) have been shown to learn dense feature representations of the nodes in a struc- ture, thus exploiting similarities between nodes and sub-trees. Also, Long-Short Term Mem- ory (Hochreiter and Schmidhuber, 1997) networks build intermediate representations of sequences, resulting in similarity estimates over sequences and their inner sub-sequences. While such methods are highly effective and reach state-of-the-art results in many tasks, their adoption can be problematic. In kernel-based Support Vector Machine (SVM) the classification model corresponds to the set of support vectors (SVs) and weights justifying the maximal margin hyperplane: the classification cost crucially de- pends on their number, as classifying a new in- stance requires …

Content truncated to 500 words