Mutual Information Alleviates Hallucinations in Abstractive Summarization

Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 602 - 631 December 7-11, 2022 ©2022 Association for Computational Linguistics UNIFIEDSKG: Unifying and Multi-Tasking Structured Knowledge Grounding with Text-to-Text Language Models Tianbao Xie∗ 1 Chen Henry Wu∗ 2 Peng Shi3 Ruiqi Zhong4 Torsten Scholak5 Michihiro Yasunaga6 Chien-Sheng Wu7 Ming Zhong8 Pengcheng Yin9 Sida I. Wang10 Victor Zhong17 Bailin Wang11 Chengzu Li12 Connor Boyle17 Ansong Ni13 Ziyu Yao14 Dragomir Radev13 Caiming Xiong7 Lingpeng Kong1,12 Rui Zhang15 Noah A. Smith16,17 Luke Zettlemoyer10,17 Tao Yu1,17 1The University of Hong Kong 2Carnegie Mellon University 3University of Waterloo 4UC Berkeley 5ServiceNow Research 6Stanford University 7Salesforce Research 8UIUC 9Google Research 10Facebook AI Research 11University of Edinburgh 12Shanghai AI Lab 13Yale University 14George Mason University 15Penn State University 16Allen Institute for Artificial Intelligence 17University of Washington Abstract Structured knowledge grounding (SKG) lever- ages structured knowledge to complete user requests, such as semantic parsing over databases and question answering over knowl- edge bases. Since the inputs and outputs of SKG tasks are heterogeneous, they have been studied separately by different communi- ties, which limits systematic and compatible research on SKG. In this paper, we overcome this limitation by proposing the UNIFIEDSKG framework, which unifies 21 SKG tasks into a text-to-text format, aiming to promote sys- tematic SKG research, instead of being exclu- sive to a single task, domain, or dataset. We use UNIFIEDSKG to benchmark T5 with dif- ferent sizes and show that T5, with simple modifications when necessary, achieves state- of-the-art performance on almost all of the 21 tasks. We further demonstrate that multi-task prefix-tuning improves the performance on most tasks, largely improving the overall per- formance. UNIFIEDSKG also facilitates the investigation of zero-shot and few-shot learn- ing, and we show that T0, GPT-3, and Codex struggle in zero-shot and few-shot learning for SKG. We also use UNIFIEDSKG to con- duct a series of controlled experiments on structured knowledge encoding variants across SKG tasks. UNIFIEDSKG is easily extensible to more tasks, and it is open-sourced at https: //github.com/hkunlp/unifiedskg.1 1 Introduction Structured knowledge (e.g., web tables, knowledge graphs, and databases) stores large amounts of data in organized structures, forming a basis for a wide range of applications, e.g., medical diagnosis, per- sonal assistants, and customer relations manage- ∗Equal contributions. Author contributions in App. A. 1Latest collections at https://unifiedskg.com. ment. Accessing and searching data in structured knowledge typically requires mastering query lan- guages through professional training. To promote the efficiency of data access, structured knowledge grounding (SKG) systems ground user requests in structured knowledge and produce various out- puts, including computer programs (e.g., SQL and SPARQL), table cell values, and natural language responses (Figure 1). For example, semantic pars- ing (Zelle and Mooney, 1996; Zettlemoyer and Collins, 2005) converts natural language questions into formal programs; knowledge-base question an- swering (Berant et al., 2013) derives answers from tables or knowledge graphs. SKG has attracted significant interest and has been studied through different tasks defined by dif- ferent communities. Recent developments in tasks, models, and datasets for …

Content truncated to 500 words