Dependency-based Mixture Language Models

Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics Volume 1: Long Papers, pages 7758 - 7773 May 22-27, 2022 c⃝2022 Association for Computational Linguistics Dependency-based Mixture Language Models Zhixian Yang and Xiaojun Wan Wangxuan Institute of Computer Technology, Peking University Center for Data Science, Peking University The MOE Key Laboratory of Computational Linguistics, Peking University yangzhixian@stu.pku.edu.cn wanxiaojun@pku.edu.cn Abstract Various models have been proposed to incor- porate knowledge of syntactic structures into neural language models. However, previous works have relied heavily on elaborate compo- nents for a specific language model, usually recurrent neural network (RNN), which makes themselves unwieldy in practice to fit into other neural language models, such as Transformer and GPT-2. In this paper, we introduce the Dependency-based Mixture Language Models. In detail, we first train neural language models with a novel dependency modeling objective to learn the probability distribution of future dependent tokens given context. We then for- mulate the next-token probability by mixing the previous dependency modeling probability distributions with self-attention. Extensive ex- periments and human evaluations show that our method can be easily and effectively applied to different neural language models while improv- ing neural text generation on various tasks.1 1 Introduction Syntactic structures serve as the principle of how words are correctly combined to form sentences. It is widely acknowledged that learning syntactic structures should improve neural text generation (Shen et al., 2018; Peng et al., 2019; Du et al., 2020). Even though current neural language mod- els, such as Transformer (Vaswani et al., 2017) and GPT-2 (Radford et al., 2019) have achieved out- standing performance without explicitly modeling latent syntactic structures, these models still fail to learn the long-range syntactic dependencies (Kun- coro et al., 2018; Xu et al., 2021). To leverage explicit syntactic knowledge in nat- ural language generation (NLG), many methods have been proposed (Wu et al., 2017; Shen et al., 2018; Zhang et al., 2019; Kim et al., 2019; Du 1Our code is available at https: //github.com/FadedCosine/ Dependency-Guided-Neural-Text-Generation et al., 2020). We conclude from previous works that knowledge of syntactic structures can bring four advantages to neural language models: (1) Syntactic structures can be modeled to obtain better representations of natural language sentences (Jacob et al., 2018; Williams et al., 2018; Wang et al., 2019). (2) Jointly training syntactic structure parsing and language modeling can contribute to each other (Shen et al., 2018; Dyer et al., 2016; Kim et al., 2019; Du et al., 2020; Shen et al., 2021b). (3) Syntactic structures can be used to directly model the composition of language (Socher et al., 2013; Casas et al., 2020) and help with the long- range dependency problem by providing shortcuts for gradient backpropagation (Chung et al., 2017). (4) Integrating syntactic structures into a neural network can improve generalization via a better in- ductive bias (Shen et al., 2019; Zhang et al., 2019). Despite these advantages, it is not trivial to in- corporate knowledge of syntactic structures into neural language models effectively and efficiently. Several practical problems …

Content truncated to 500 words