Combining Deep Generative Models and Multi-lingual Pretraining for Semi-supervised Document Classification

Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics, pages 894–908 April 19 - 23, 2021. ©2021 Association for Computational Linguistics 894 Combining Deep Generative Models and Multi-lingual Pretraining for Semi-supervised Document Classification Yi Zhu♥ Ehsan Shareghi♠♥ Yingzhen Li♦∗ Roi Reichart♣ Anna Korhonen♥ ♥ Language Technology Lab, University of Cambridge ♠ Department of Data Science & AI, Monash University ♦ Department of Computing, Imperial College London ♣ Faculty of Industrial Engineering and Management, Technion, IIT {yz568,alk23}@cam.ac.uk, ehsan.shareghi@monash.edu yingzhen.li@imperial.ac.uk, roiri@technion.ac.il Abstract Semi-supervised learning through deep gen- erative models and multi-lingual pretraining techniques have orchestrated tremendous suc- cess across different areas of NLP. Nonethe- less, their development has happened in isola- tion, while the combination of both could po- tentially be effective for tackling task-specific labelled data shortage. To bridge this gap, we combine semi-supervised deep genera- tive models and multi-lingual pretraining to form a pipeline for document classification task. Compared to strong supervised learning baselines, our semi-supervised classification framework is highly competitive and outper- forms the state-of-the-art counterparts in low- resource settings across several languages. 1 1 Introduction Multi-lingual pretraining has been shown to effec- tively use unlabelled data through learning shared representations across languages that can be trans- ferred to downstream tasks (Artetxe and Schwenk, 2019; Devlin et al., 2019; Wu and Dredze, 2019; Conneau and Lample, 2019). Nonetheless, the lack of labelled data still leads to inferior perfor- mance of the same model compared to those trained in languages with more labelled data such as En- glish (Zeman et al., 2018; Zhu et al., 2019). Semi-supervised learning is another appealing paradigm that supplements the labelled data with unlabelled data which is easy to acquire (Blum and Mitchell, 1998; Zhou and Li, 2005; McClosky et al., 2006, inter alia). In particular, deep generative models (DGMs) such as variational autoencoder (VAE; Kingma and Welling (2014)) are capable of capturing complex data distributions at scale with rich latent representations, and they have been used ∗Work done while at Microsoft Research Cambridge. 1Code is available at https://github.com/ cambridgeltl/mling_sdgms. for semi-supervised learning in various tasks in NLP (Xu et al., 2017; Yin et al., 2018; Choi et al., 2019; Xie and Ma, 2019), as well as inducing cross- lingual word embeddings (Wei and Deng, 2017), and representation learning in combination with Transformers via pretraining (Li et al., 2020). To leverage the benefits of both worlds, we propose a pipeline method by combining semi- supervised DGMs (SDGMs) based on M1+M2 model (Kingma et al., 2014) with multi-lingual pretraining. The pretrained model serves as multi- lingual encoder, and SDGMs can operate on top of it independently of encoding architecture. To high- light such independence, we experiment with two pretraining settings: (1) our LSTM-based cross- lingual VAE, and (2) the current stat-of-the-art (SOTA) multi-lingual BERT (Devlin et al., 2019). Our experiments on document classification in several languages show promising results via the SDGM framework with different encoders, outperforming the SOTA supervised counterparts. We also illustrate that the end-to-end training of M1+M2 …

Content truncated to 500 words