Latent-Variable Generative Models for Data-Efficient Text Classification

Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing, pages 507–517, Hong Kong, China, November 3–7, 2019. c⃝2019 Association for Computational Linguistics 507 Latent-Variable Generative Models for Data-Efficient Text Classification Xiaoan Ding1 Kevin Gimpel2 1University of Chicago, Chicago, IL, 60637, USA 2Toyota Technological Institute at Chicago, Chicago, IL, 60637, USA xiaoanding@uchicago.edu, kgimpel@ttic.edu Abstract Generative classifiers offer potential advan- tages over their discriminative counterparts, namely in the areas of data efficiency, ro- bustness to data shift and adversarial exam- ples, and zero-shot learning (Ng and Jordan, 2002; Yogatama et al., 2017; Lewis and Fan, 2019). In this paper, we improve generative text classifiers by introducing discrete latent variables into the generative story, and explore several graphical model configurations. We parameterize the distributions using standard neural architectures used in conditional lan- guage modeling and perform learning by di- rectly maximizing the log marginal likelihood via gradient-based optimization, which avoids the need to do expectation-maximization. We empirically characterize the performance of our models on six text classification datasets. The choice of where to include the latent vari- able has a significant impact on performance, with the strongest results obtained when using the latent variable as an auxiliary condition- ing variable in the generation of the textual in- put. This model consistently outperforms both the generative and discriminative classifiers in small-data settings. We analyze our model by using it for controlled generation, finding that the latent variable captures interpretable prop- erties of the data, even with very small training sets. 1 Introduction The most widely-used neural network classifiers are discriminative, that is, they are trained to ex- plicitly favor the gold standard label over others. The alternative is to design classifiers that are gen- erative; these follow a generative story that in- cludes predicting the label and then the data condi- tioned on the label. Discriminative classifiers are preferred because they generally outperform their generative counterparts on standard benchmarks. These benchmarks typically assume large anno- tated training sets, little mismatch between train- ing and test distributions, relatively clean data, and a lack of adversarial examples (Zue et al., 1990; Marcus et al., 1993; Deng et al., 2009; Lin et al., 2014). However, when conditions are not ideal for dis- criminative classifiers, generative classifiers can actually perform better. Ng and Jordan (2002) showed theoretically that linear generative classi- fiers approach their asymptotic error rates more rapidly than discriminative ones. Based on this finding, Yogatama et al. (2017) empirically char- acterized the performance of RNN-based gen- erative classifiers, showing advantages in sam- ple complexity, zero-shot learning, and continual learning. Recent work in generative question an- swering models (Lewis and Fan, 2019) demon- strates better robustness to biased training data and adversarial testing data than state-of-the-art dis- criminative models. In this paper, we focus on settings with small amounts of annotated data and improve genera- tive text classifiers by introducing discrete latent variables into the generative story. Accordingly, the training objective is changed to …

Content truncated to 500 words