Learning Structural Kernels for Natural Language Processing

Learning Structural Kernels for Natural Language Processing Daniel Beck Department of Computer Science University of Sheffield, United Kingdom debeck1@sheffield.ac.uk Trevor Cohn Computing and Information Systems University of Melbourne, Australia t.cohn@unimelb.edu.au Christian Hardmeier Department of Linguistics and Philology Uppsala University, Sweden christian.hardmeier@lingfil.uu.se Lucia Specia Department of Computer Science University of Sheffield, United Kingdom l.specia@sheffield.ac.uk Abstract Structural kernels are a flexible learning paradigm that has been widely used in Natural Language Processing. However, the problem of model selection in kernel-based methods is usually overlooked. Previous approaches mostly rely on setting default values for ker- nel hyperparameters or using grid search, which is slow and coarse-grained. In con- trast, Bayesian methods allow efficient model selection by maximizing the evidence on the training data through gradient-based methods. In this paper we show how to perform this in the context of structural kernels by using Gaussian Processes. Experimental results on tree kernels show that this procedure results in better prediction performance compared to hyperparameter optimization via grid search. The framework proposed in this paper can be adapted to other structures besides trees, e.g., strings and graphs, thereby extending the util- ity of kernel-based methods. 1 Introduction Kernel-based methods are a staple machine learning approach in Natural Language Processing (NLP). Frequentist kernel methods like the Support Vector Machine (SVM) pushed the state of the art in many NLP tasks, especially classification and regression. One interesting aspect of kernels is their ability to be defined directly on structured objects like strings, trees and graphs. This approach has the potential to move the modelling effort from feature engineering to kernel engineering. This is useful when we do not have much prior knowledge about how the data behaves, as we can more readily define a similarity metric between inputs instead of trying to character- ize which features are the best for the task at hand. Kernels are a very flexible framework: they can be combined and parameterized in many different ways. Complex kernels, however, lead to the prob- lem of model selection, where the aim is to obtain the best kernel configuration in terms of hyperpa- rameter values. The usual approach for model selec- tion in frequentist methods is to employ grid search on some development data disjoint from the training data. This approach can rapidly become impracti- cal when using complex kernels which increase the number of model hyperparameters. Grid search also requires the user to explicitly set the grid values, making it difficult to fine tune the hyperparameters. Recent advances in model selection tackle some of these issues, but have several limitations (see §6 for details). Our proposed approach for model selection re- lies on Gaussian Processes (GPs) (Rasmussen and Williams, 2006), a widely used Bayesian kernel ma- chine. GPs allow efficient and fine-grained model selection by maximizing the evidence on the training data using gradient-based methods, dropping the re- quirement for development data. As a Bayesian pro- cedure, GPs also naturally balance between model capacity and generalization. GPs have been shown to achieve state of the …

Content truncated to 500 words