TegTok: Augmenting Text Generation via Task-specific and Open-world Knowledge

Findings of the Association for Computational Linguistics: ACL 2022, pages 1597 - 1609 May 22-27, 2022 c⃝2022 Association for Computational Linguistics TEGTOK: Augmenting Text Generation via Task-specific and Open-world Knowledge Chao-Hong Tan1∗, Jia-Chen Gu1∗, Chongyang Tao2, Zhen-Hua Ling1†, Can Xu2, Huang Hu2, Xiubo Geng2, Daxin Jiang2† 1National Engineering Research Center for Speech and Language Information Processing, University of Science and Technology of China, Hefei, China 2Microsoft, Beijing, China {chtan,gujc}@mail.ustc.edu.cn, zhling@ustc.edu.cn, {chotao,caxu,huahu,xigeng,djiang}@microsoft.com Abstract Generating natural and informative texts has been a long-standing problem in NLP. Much effort has been dedicated into incorporating pre- trained language models (PLMs) with various open-world knowledge, such as knowledge graphs or wiki pages. However, their ability to access and manipulate the task-specific knowl- edge is still limited on downstream tasks, as this type of knowledge is usually not well covered in PLMs and is hard to acquire. To address the problem, we propose augmenting TExt Generation via Task-specific and Open-world Knowledge (TEGTOK) in a unified framework. Our model selects knowledge entries from two types of knowledge sources through dense retrieval and then injects them into the input en- coding and output decoding stages respectively on the basis of PLMs. With the help of these two types of knowledge, our model can learn what and how to generate. Experiments on two text generation tasks of dialogue generation and question generation, and on two datasets show that our method achieves better performance than various baseline models. 1 Introduction Enabling natural models to generate natural and in- formative sequences is a challenging yet intriguing problem of artificial intelligence and has attracted increasing attention due to its promising potentials and alluring commercial values (Bahdanau et al., 2015; Du et al., 2017; Kepuska and Bohouta, 2018; Berdasco et al., 2019; Zhou et al., 2020; Gehrmann et al., 2021). Thanks to the achievements on neural sequence modeling and pre-training technologies, current generative models are able to generate nature and fluency target sequences using either encoder-decoder architectures (Sutskever et al., 2014; Bahdanau et al., 2015; Vaswani et al., 2017) or language models (Radford et al., 2019; Brown ∗Work done during the internship at Microsoft. †Corresponding author. et al., 2020; Lewis et al., 2020a) Despite these methods being the state-of-the-art frameworks for NLG, they are often provided limited knowledge to generate the desired output. Thus, the performance of text generation is still far from satisfaction in many real-world scenarios (Yu et al., 2020a). Recently, much effort has been dedicated into incorporating traditional generative models or pre- trained language models (PLMs) with a variety of open-world knowledge, such as structural knowl- edge bases (e.g., ConceptNet) (Speer and Havasi, 2012; Speer et al., 2017) or unstructured documents (e.g., documents from Wikipedia) (Zhou et al., 2018c; Dinan et al., 2019). By providing the supplementary knowledge of an entity mentioned within or the background knowledge of a source text, it can help to better understand the input text and its surrounding context, and to ameliorate the informativeness of the generated text. Although the open-world knowledge brings improvement to …

Content truncated to 500 words