Evaluation of Deep Gaussian Processes for Text Classification

Proceedings of the 12th Conference on Language Resources and Evaluation (LREC 2020), pages 1485–1491 Marseille, 11–16 May 2020 c⃝ European Language Resources Association (ELRA), licensed under CC-BY-NC 1485 Evaluation of Deep Gaussian Processes for Text Classification P. Jayashree, P. K. Srijith Department of Computer Science and Engineering IIT Hyderabad {cs16resch11002, srijith}@iith.ac.in Abstract With the tremendous success of deep learning models on computer vision tasks, there are various emerging works on the Natural Language Processing (NLP) task of Text Classification using parametric models. However, it constrains the expressability limit of the function and demands enormous empirical efforts to come up with a robust model architecture. Also, the huge parameters involved in the model causes over-fitting when dealing with small datasets. Deep Gaussian Processes (DGP) offer a Bayesian non-parametric modelling framework with strong function compositionality, and helps in overcoming these limitations. In this paper, we propose DGP models for the task of Text Classification and an empirical comparison of the performance of shallow and Deep Gaussian Process models is made. Extensive experimentation is performed on the benchmark Text Classification datasets such as TREC (Text REtrieval Conference), SST (Stanford Sentiment Treebank), MR (Movie Reviews), R8 (Reuters-8), which demonstrate the effectiveness of DGP models. Keywords: Text Classification, Bayesian deep learning, Gaussian Process, Convolutional Gaussian Process. 1. Introduction and Related Work Text classification is a primary task in NLP which helps in solving multifold problems such as sentiment analysis (Le and Mikolov, 2014, Socher et al., 2013), spam detection (Wang, 2010) etc. Recently, deep learning models have obtained significant performance in computer vision. Inspired by which, there are some works on Deep learning models for NLP tasks (Kim, 2014, Yih et al., 2014, Shen et al., 2014, Kalchbrenner et al., 2014). However, deep learning models involve millions of learnable parameters, requires large dataset, does not make any uncertainty estimates, and incurs a tedious model selection procedure. Gaussian Processes (GPs) offers a Bayesian non-parametric alternative framework to the existing parametric models. (Neal, 1994) discusses the equivalence between a bayesian neural network (with single hidden layer having infinite hidden units) and gaussian process. As the number of units in the hidden layer tend to infinity, the network converges to a gaussian process. The stacking of GPs lead to a Deep Gaussian Process (DGP) model (Damianou and Lawrence, 2013a). That is, a model where the observations are modelled as the output of the GP whose input is facilitated by another GP is DGP and it helps to overcome the aforementioned limitations. There are many variants of DGP models with emphasis on the inferencing techniques used. (Damianou and Lawrence, 2013a) discusses on mean field variational posterior over hidden layers with factorised form as in (Titsias and Lawrence, 2010) and (Hensman and Lawrence, 2014) presents nested variational inference approach . (Dai et al., 2016) discusses on amortized inference whereas (Bui et al., 2016) uses approximate Expectation Propagation approach. (Salimbeni and Deisenroth, 2017) extends (Damianou and Lawrence, 2013a) with variational poste- rior conditioned on the previous layer, thereby facilitating parallelization of …

Content truncated to 500 words