How do decoding algorithms distribute information in dialogue responses?

Findings of the Association for Computational Linguistics: EACL 2023, pages 953–962 May 2-6, 2023 ©2023 Association for Computational Linguistics How do decoding algorithms distribute information in dialogue responses? Saranya Venkatraman Pennsylvania State University saranyav@psu.edu He He New York University hhe@nyu.edu David Reitter Google Research reitter@google.com Abstract Humans tend to follow the Uniform Informa- tion Density (UID) principle by distributing information evenly in utterances. We study if decoding algorithms implicitly follow this UID principle, and under what conditions ad- herence to UID might be desirable for dia- logue generation. We generate responses us- ing different decoding algorithms with GPT- 2 on the Persona-Chat dataset and collect hu- man judgments on their quality using Ama- zon Mechanical Turk. We find that (i) surpris- ingly, model-generated responses follow the UID principle to a greater extent than human responses, and (ii) decoding algorithms that promote UID do not generate higher-quality responses. Instead, when we control for sur- prisal, non-uniformity of information density correlates with the quality of responses with very low/high surprisal. Our findings indi- cate that encouraging non-uniform responses is a potential solution to the “likelihood trap” problem (quality degradation in very high- likelihood text). Our dataset containing mul- tiple candidate responses per dialog history along with human-annotated quality ratings is available at: https://huggingface.co/ datasets/saranya132/dialog_uid_gpt2. 1 Introduction The Uniform Information Density (UID) hypoth- esis states that humans distribute information in their utterances evenly for optimal communica- tion (Jaeger, 2010; Fenk and Fenk, 1980). Con- sequently, language generation has benefitted from UID-based objectives and regularization (Meister et al., 2022; Wei et al., 2021). Specifically, Meis- ter et al. (2020) argued that UID can be optimized for machine translation using beam search. Yet, the effect of different decoding algorithms on in- formation density distributions of generated text are unknown, as is UID’s broader role in neural response generation in the special case of dialogue Dialog History Speaker A: hello, how are you today? Speaker A: that is cool! i just like to skateboard! Speaker B: good. yourself? listening to classical music. Speaker B: CANDIDATE RESPONSES Human Annotation Scores Candidate Responses can you do a kick flip haha. i love skating too, but i have a truck to keep as well. nice. i love to do stuff like that. i also swim. are you a skater or do you do it daily? Type Reference Text Nucleus sampling Top-k Greedy Related {3,3,3} {.,.,.} {.,.,.} {.,.,.} Furthering {3,3,3} {.,.,.} {.,.,.} {.,.,.} Interesting {3,2,2} {.,.,.} {.,.,.} {.,.,.} Figure 1: Our dataset contains 4 candidate responses for every dialog history, along with human annotations for 3 qualitative measures. models. Here, we investigate (i) if different decod- ing algorithms follow the UID principle, and (ii) if following the UID principle is beneficial for dia- logue response generation, and (iii) collect human annotations of qualitative measures for multiple candidate responses to dialog histories generated using different decoding algorithms (Figure 1) to study the relationship of dialog response quality and UID. We operationalize UID as the variance of surprisal and measure its correlation …

Content truncated to 500 words