Locally Typical Sampling

Locally Typical Sampling Clara Meister1 Tiago Pimentel2 Gian Wiher1 Ryan Cotterell1,2 1ETH Z¨urich, Switzerland 2University of Cambridge, UK clara.meister@inf.ethz.ch tp472@cam.ac.uk gian.wiher@inf.ethz.ch ryan.cotterell@inf.ethz.ch Abstract Today’s probabilistic language generators fall short when it comes to producing coherent and fluent text despite the fact that the underlying models perform well under standard metrics (e.g., perplexity). This discrepancy has puzzled the language generation community for the last few years. In this work, we posit that the ab- straction of natural language generation as a discrete stochastic process—which allows for an information-theoretic analysis—can pro- vide new insights into the behavior of probabi- listic language generators, for example, why high-probability texts can be dull or repetitive. Humans use language as a means of com- municating information, aiming to do so in a simultaneously efficient and error-minimizing manner; in fact, psycholinguistics research suggests humans choose each word in a string with this subconscious goal in mind. We for- mally define the set of strings that meet this criterion: Those for which each word has an information content close to the expected in- formation content, namely, the conditional entropy of our model. We then propose a sim- ple and efficient procedure for enforcing this criterion when generating from probabilistic models, which we call locally typical sam- pling. Automatic and human evaluations show that, in comparison to nucleus and top-k sam- pling, locally typical sampling offers com- petitive performance (in both abstractive summarization and story generation) in terms of quality while consistently reducing degen- erate repetitions. 1 Introduction Modern probabilistic models have repeatedly demonstrated their prowess at modeling natural language, placing high probability on held-out corpora from many different domains (Brown et al., 2020; Hoffmann et al., 2022; Chowdhery et al., 2022). Yet when used as text generators, their performance is far from perfect. One of the largest determinants of the generated text’s qual- ity is the choice of decoding strategy—that is, the decision rule used to extract strings from a model. Perhaps surprisingly, for many language generation tasks, decoding strategies that aim to find the highest-probability strings produce text that is undesirable (Holtzman et al., 2020; See et al., 2019; Eikema and Aziz, 2020; Zhang et al., 2021; DeLucia et al., 2021). For instance, Stahlberg and Byrne (2019) report that in their neu- ral machine translation experiments, the highest- probability string is usually the empty string. On the other hand, stochastic strategies, which take random samples from the model, often lead to text with better qualitative properties (Fan et al., 2018; Holtzman et al., 2020; Basu et al., 2021). How- ever, stochastic strategies still have a host of other problems, while not entirely dispensing with those seen in maximization-based approaches.1 At first glance, it is unintuitive that high- probability strings are often neither desirable nor human-like. Due to this pathology, a number of studies have concluded that there must be faults in the training objective or architecture of the probabilistic models behind language generators (Welleck et al., 2020; Guan et al., 2020; Li et al., 2020, inter alia). …

Content truncated to 500 words