EnsLM: Ensemble Language Model for Data Diversity by Semantic Clustering

Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing, pages 2954–2967 August 1–6, 2021. ©2021 Association for Computational Linguistics 2954 EnsLM: Ensemble Language Model for Data Diversity by Semantic Clustering Zhibin Duan*1, Hao Zhang*†1, Chaojie Wang1, Zhengjue Wang1, Bo Chen†1, Mingyuan Zhou2 1National Laboratory of Radar Signal Processing, Xidian University, Xi’an, China 2McCombs School of Business The University of Texas at Austin, Austin, TX 78712, USA {xd zhibin, zhanghao xidian}@163.com bchen@mail.xidian.edu.cn, Mingyuan.Zhou@mccombs.utexas.edu Abstract Natural language processing often faces the problem of data diversity such as different domains, themes, styles and so on. There- fore, a single language model (LM) is insuffi- cient to learn all knowledge from diverse sam- ples. To solve this problem, we firstly pro- pose an autoencoding topic model with mix- ture prior (mATM) to perform clustering for the data, where the clusters defined in seman- tic space describe the data diversity. Having obtained the clustering assignment for each sample, we develop the ensemble LM (En- sLM) with the technique of weight modula- tion. Specifically, EnsLM contains a backbone which is adjusted by a few modulated weights to fit for different sample clusters. As a re- sult, the backbone learns the shared knowledge among all clusters while modulated weights extract the cluster-specific features. EnsLM can be trained jointly with mATM with flexi- ble LM backbone. We evaluate the effective- ness of both mATM and EnsLM on different language understanding and generative tasks. 1 Introduction It is common knowledge in modern natural lan- guage processing (NLP) that natural language varies greatly across domains, themes, styles, gen- res and many other linguistic nuances (Van der Wees et al., 2015; van der Wees, 2017; Niu et al., 2017). Generally, we call such nature of language as data diversity. Many existing works (Liu et al., 2017; Cai and Wan, 2019; Hu et al., 2019) have illustrated that data diversity will affect the perfor- mance of LMs if we just train a single LM over the entire dataset, even though fine-tuning a pre- trained LM (that has been pre-training on a very large corpus) such as Bert (Devlin et al., 2019) on current task (Aharoni and Goldberg, 2020). * Equal contribution. † Corresponding author. (a) LDA (b) mATM Figure 1: The distribution of samples on seman- tic space on 4 domains (different products) of Ama- zon dataset. The sample clustering characteristics of mATM can reflect the data diversity (domain in this ex- ample) in the corpus. The domain diversity in dataset is a very com- mon type of data diversity. In some cases, if we can obtain a well-defined domain label for each sample, some works (Jiang et al., 2020; Du et al., 2020; Wright and Augenstein, 2020) try to consider the multi-domain property of data in developing the LMs. However, these pre-defined domain labels are not always accurate or even available (Aharoni and Goldberg, 2020), especially for the wild datasets, in which data come from different sources, such …

Content truncated to 500 words