Focused Attention Improves Document-Grounded Generation

Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 4274–4287 June 6–11, 2021. ©2021 Association for Computational Linguistics 4274 Focused Attention Improves Document-Grounded Generation Shrimai Prabhumoye1∗, Kazuma Hashimoto2, Yingbo Zhou2, Alan W Black1, Ruslan Salakhutdinov1 1Carnegie Mellon University, 2Salesforce Research {sprabhum@cs.cmu.edu, k.hashimoto@salesforce.com} Abstract Document grounded generation is the task of using the information provided in a document to improve text generation. This work focuses on two different document grounded genera- tion tasks: Wikipedia Update Generation task and Dialogue response generation. Our work introduces two novel adaptations of large scale pre-trained encoder-decoder models focusing on building context driven representation of the document and enabling specific attention to the information in the document. Addition- ally, we provide a stronger BART baseline for these tasks. Our proposed techniques outper- form existing methods on both automated (at least 48% increase in BLEU-4 points) and hu- man evaluation for closeness to reference and relevance to the document. Furthermore, we perform comprehensive manual inspection of the generated output and categorize errors to provide insights into future directions in mod- eling these tasks. 1 Introduction Natural language generation (NLG) systems are increasingly expected to be naturalistic, content- ful, and situation-aware due to their popularity and pervasiveness in human life (Reiter and Dale, 2000; Mitchell et al., 2014). This is particularly relevant in dialogue systems (Zhang et al., 2018a; Niu and Bansal, 2018), machine translation sys- tems (Mirkin and Meunier, 2015; Rabinovich et al., 2017), story generation (Fan et al., 2018; Yao et al., 2019), and question answering systems (Gatius, 2017; Reddy et al., 2019). Despite these mainstream applications, NLG sys- tems face the challenges of being bland, devoid of content, generating generic outputs and hallucinat- ing information (Wiseman et al., 2017; Li et al., 2016; Holtzman et al., 2020; Welleck et al., 2020). Grounding the generation in different modalities ∗ Work done during internship at Salesforce. like images (Huang et al., 2016; Mostafazadeh et al., 2017; Shuster et al., 2018), videos (Palaskar et al., 2019; Regneri et al., 2013), and structured data (Banik et al., 2013; Gardent et al., 2017) al- leviates some of these issues. Generating natural language from schematized or structured data such as database records, slot-value pair, and Wikipedia Infobox has been explored in prior work (Mei et al., 2016; Wen et al., 2015; Lebret et al., 2016). Al- though useful, these tasks encounter difficulties such as general applicability (databases may not be available for all domains) and are constrained by the available resources (size of the database). Document grounded generation mitigates these applicability issues by exploiting the vast avail- ability of data in unstructured form (e.g. books, encyclopedias, news articles, and Wikipedia arti- cles). This enhances the applicability of document grounded generation to a wide range of domains with limited (or no) availability of structured data. Hence, recent work has focused on defining new tasks and carving the scope of the problems (Liu et al., 2018; Prabhumoye et al., 2019; Faltings et …

Content truncated to 500 words