Counter-Argument Generation by Attacking Weak Premises

Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021, pages 175–185 August 1–6, 2021. ©2021 Association for Computational Linguistics 175 Discovering Topics in Long-tailed Corpora with Causal Intervention Xiaobao Wu1 Chunping Li1 Yishu Miao2 1Tsinghua University wxb18@mails.tsinghua.edu.cn,2Imperial College London cli@mail.tsinghua.edu.cn, y.miao20@imperial.ac.uk Abstract Topic models are effective in capturing the la- tent semantics of large-scale textual data while existing methods are normally designed and evaluated on balanced corpora. However, it contradicts the fact that general corpora in our world are naturally long-tailed, and the long- tailed bias can highly impair the topic model- ing performance. Therefore, in this paper, we propose a causal inference framework to ex- plain and overcome the issues of topic mod- eling on long-tailed corpora. In a neat and elegant way, causal intervention is applied in training to take out the influence brought by the long-tailed bias. Extensive experiments on manually constructed and naturally collected datasets demonstrate that our model can miti- gate the bias effect, greatly improve topic qual- ity and better discover the hidden semantics on the tail. 1 Introduction Topic models are proposed to discover the under- lying topics and semantic structures from unla- belled text collections. Due to the effectiveness and interpretability, topic models have been ap- plied in various downstream tasks like information retrieval (Wang et al., 2007), content summariza- tion (Ma et al., 2012) and recommendation systems (McAuley and Leskovec, 2013). One of the most widely used topic models is Latent Dirichlet Al- location (LDA) (Blei et al., 2003), a probabilistic graphical model using the conjugate of Dirichlet and Multinomial distribution and inferring the pa- rameters with approximation methods(Griffiths and Steyvers, 2004; Blei et al., 2017). Recently, some popular neural topic models based on Variational AutoEncoder (VAE) (Kingma and Welling, 2014; Rezende et al., 2014) have been introduced, such as Neural Variational Document Model (NVDM) (Miao et al., 2016) and Product of Experts LDA (ProdLDA) (Srivastava and Sutton, 2017). Com- pared to probabilistic ones, they can easily carry out the inference by gradient backpropagation. However, these topic models are generally de- signed and evaluated on balanced corpora, such as the commonly used 20News (Lang, 1995) with evenly distributed labels through which we can in- fer that the latent topics are also evenly distributed. It hence conflicts with the fact that natural text col- lections are regularly long-tail distributed follow- ing Zipf’s law (Reed, 2001), especially the textual data on social network platforms(Zhang and Luo, 2019). More precisely, Figure 1 illustrates that in a collected corpus, documents about some hot topics are numerous (head topics), while the documents about most topics are few (tail topics). Due to this bias, similar to long-tailed classification tasks where a classifier favors to predict an image as the head classes (Kang et al., 2020; Zhou et al., 2020), topic models on long-tailed corpora tend to reveal the semantics of documents about head topics and ignore the documents about tail topics to a great extent. Namely, the discovered topics are mostly about the latent head ones in …

Content truncated to 500 words