GLUECoS: An Evaluation Benchmark for Code-Switched NLP

Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 340–350 July 5 - 10, 2020. c⃝2020 Association for Computational Linguistics 340 Neural Topic Modeling with Bidirectional Adversarial Training Rui Wang† Xuemeng Hu† Deyu Zhou†∗ Yulan He§ Yuxuan Xiong† Chenchen Ye† Haiyang Xu‡ †School of Computer Science and Engineering, Key Laboratory of Computer Network and Information Integration, Ministry of Education, Southeast University, China §Department of Computer Science, University of Warwick, UK ‡AI Labs - Didi Chuxing Co., Ltd. - Beijing, China {rui wang, xuemenghu, d.zhou, yuxuanxiong, chenchenye}@seu.edu.cn, yulan.he@warwick.ac.uk, xuhaiyangsnow@didiglobal.com Abstract Recent years have witnessed a surge of inter- ests of using neural topic models for automatic topic extraction from text, since they avoid the complicated mathematical derivations for model inference as in traditional topic mod- els such as Latent Dirichlet Allocation (LDA). However, these models either typically assume improper prior (e.g. Gaussian or Logistic Nor- mal) over latent topic space or could not infer topic distribution for a given document. To address these limitations, we propose a neu- ral topic modeling approach, called Bidirec- tional Adversarial Topic (BAT) model, which represents the first attempt of applying bidi- rectional adversarial training for neural topic modeling. The proposed BAT builds a two- way projection between the document-topic distribution and the document-word distribu- tion. It uses a generator to capture the se- mantic patterns from texts and an encoder for topic inference. Furthermore, to incorpo- rate word relatedness information, the Bidirec- tional Adversarial Topic model with Gaussian (Gaussian-BAT) is extended from BAT. To ver- ify the effectiveness of BAT and Gaussian- BAT, three benchmark corpora are used in our experiments. The experimental results show that BAT and Gaussian-BAT obtain more co- herent topics, outperforming several compet- itive baselines. Moreover, when performing text clustering based on the extracted topics, our models outperform all the baselines, with more significant improvements achieved by Gaussian-BAT where an increase of near 6% is observed in accuracy. 1 Introduction Topic models have been extensively explored in the Natural Language Processing (NLP) commu- nity for unsupervised knowledge discovery. Latent Dirichlet Allocation (LDA) (Blei et al., 2003), the ∗corresponding author Logistic-Normal Dirichlet ¡ + Figure 1: Illustrated probability simplex with Logistic- Normal distribution and Dirichlet distribution. most popular topic model, has been extended (Lin and He, 2009; Zhou et al., 2014; Cheng et al., 2014) for various extraction tasks. Due to the difficulty of exact inference, most LDA variants require ap- proximate inference methods, such as mean-field methods and collapsed Gibbs sampling. However, these approximate approaches have the drawback that small changes to the modeling assumptions result in a re-derivation of the inference algorithm, which can be mathematically arduous. One possible way in addressing this limitation is through neural topic models which employ black- box inference mechanism with neural networks. In- spired by variational autoencoder (VAE) (Kingma and Welling, 2013), Srivastava and Sutton (2017) used the Logistic-Normal prior to mimic the sim- plex in latent topic space and proposed the Neu- ral Variational LDA (NVLDA). Moreover, they replaced …

Content truncated to 500 words