Refocusing on Relevance: Personalization in NLG

Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 5190–5202 November 7–11, 2021. c⃝2021 Association for Computational Linguistics 5190 Refocusing on Relevance: Personalization in NLG Shiran Dudy Department of Computer Science University of Colorado shdu9019@colorado.edu Steven Bedrick Department of Medical Informatics and Clinical Epidemiology Oregon Health & Science University bedricks@ohsu.edu Bonnie Webber Institute for Language, Cognition and Computation School of Informatics University of Edinburgh bonnie.webber@ed.ac.uk Abstract Many NLG tasks such as summarization, dia- logue response, or open domain question an- swering focus primarily on a source text in or- der to generate a target response. This stan- dard approach falls short, however, when a user’s intent or context of work is not easily recoverable based solely on that source text– a scenario that we argue is more of the rule than the exception. In this work, we argue that NLG systems in general should place a much higher level of emphasis on making use of ad- ditional context, and suggest that relevance (as used in Information Retrieval) be thought of as a crucial tool for designing user-oriented text-generating tasks. We further discuss pos- sible harms and hazards around such person- alization, and argue that value-sensitive design represents a crucial path forward through these challenges. 1 Introduction The evaluation of natural language generation tasks (such as automatic summarization, machine transla- tion, and dialogue generation) is commonly framed as one of comparing an automated system’s gen- erated output to some reference output,1 with the goal of achieving as close an alignment as possible. Implicit in this experimental framing is the notion that for any given system input, there must exist a single, “correct” output. While this may arguably be a necessary simpli- fying assumption in terms of experimental evalu- ation,2 this “one-size-fits-all” philosophy has also constrained the ways in which NLG systems are designed, trained, and deployed. For example, stan- dard approaches to automated document summa- rization and machine translation rely only on the 1In some cases, a small set of reference outputs may be used, as in BLEU’s original formulation. 2Statistical metrics based on this assumption often fail to reflect human judgments of quality or performance; see Novikova et al.’s recent analysis. original source text, and do not typically take into consideration contextual factors involving the user or their situation of use. The problem is even larger when we consider NLG systems whose outputs are less firmly grounded to some specific input text, such as dialogue and question-answering systems. This has a number of negative effects on NLG system performance and utility. One such effect is system output that does not meet a given user’s needs: for example, a summarization system that chooses to focus on different aspects of the source text than those in which the user was interested, or a machine translation system whose output is of an inappropriate register of formality. Other ef- fects are more subtle; for example, consider that, by designing systems to produce a single universal output, we amplify the effects …

Content truncated to 500 words