Discrete Latent Variable Representations for Low-Resource Text Classification

Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 4831–4842 July 5 - 10, 2020. c⃝2020 Association for Computational Linguistics 4831 Discrete Latent Variable Representations for Low-Resource Text Classification Shuning Jin1∗ Sam Wiseman2 Karl Stratos1 Karen Livescu2 1Rutgers University 2Toyota Technological Institute at Chicago shuning.jin@rutgers.edu, {swiseman,klivescu}@ttic.edu, stratos@cs.rutgers.edu Abstract While much work on deep latent variable models of text uses continuous latent vari- ables, discrete latent variables are interesting because they are more interpretable and typi- cally more space efficient. We consider sev- eral approaches to learning discrete latent vari- able models for text in the case where ex- act marginalization over these variables is in- tractable. We compare the performance of the learned representations as features for low- resource document and sentence classification. Our best models outperform the previous best reported results with continuous representa- tions in these low-resource settings, while learning significantly more compressed repre- sentations. Interestingly, we find that an amor- tized variant of Hard EM performs particularly well in the lowest-resource regimes.1 1 Introduction Deep generative models with latent variables have become a major focus of NLP research over the past several years. These models have been used both for generating text (Bowman et al., 2016) and as a way of learning latent representations of text for downstream tasks (Yang et al., 2017; Gururan- gan et al., 2019). Most of this work has modeled the latent variables as being continuous, that is, as vectors in Rd, in part due to the simplicity of per- forming inference over (certain) continuous latents using variational autoencoders and the reparameter- ization trick (Kingma and Welling, 2014; Rezende et al., 2014). At the same time, deep generative models with discrete latent variables are attractive because the latents are arguably more interpretable, and be- cause they lead to significantly more compressed ∗Work done as an intern at Toyota Technological Institute at Chicago. 1Code available on GitHub: https://github.com/ shuningjin/discrete-text-rep representations: A representation consisting of M floating point values conventionally requires M × 32 bits, whereas M integers in {1, . . . , K} requires only M × log2 K bits. Unfortunately, discrete latent variable models have a reputation for being more difficult to learn. We conduct a thorough comparison of several pop- ular methods for learning such models, all within the framework of maximizing the evidence lower bound (ELBO) on the training data. In particular, we compare learning such models either with a Vector Quantized-VAE (van den Oord et al., 2017, VQ-VAE), a more conventional VAE with discrete latent variables (Jang et al., 2017; Maddison et al., 2017), or with an amortized version of “Hard” or “Viterbi” Expectation Maximization (Brown et al., 1993), which to our knowledge has not been ex- plored to date. We consider both models where the latents are local (i.e., per token) and where they are global (i.e., per sentence); we assess the quality of these learned discrete representations as features for a low-resource text classifier, as suggested by Gururangan et al. (2019), and in …

Content truncated to 500 words