Script Normalization for Unconventional Writing of Under-Resourced Languages in Bilingual Communities

Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics Volume 1: Long Papers, pages 1437–1455 July 9-14, 2023 ©2023 Association for Computational Linguistics On the Efficacy of Sampling Adapters Clara Meister Tiago Pimentel Luca Malagutti Ethan G. Wilcox Ryan Cotterell ETH Zürich University of Cambridge meistecl@inf.ethz.ch tp472@cam.ac.uk lmalagutti@inf.ethz.ch ethan.wilcox@inf.ethz.ch ryan.cotterell@inf.ethz.ch Abstract Sampling is a common strategy for generating text from probabilistic models, yet standard ancestral sampling often results in text that is incoherent or ungrammatical. To alleviate this issue, various modifications to a model’s sampling distribution, such as nucleus or top-k sampling, have been introduced and are now ubiquitously used in language generation systems. We propose a unified framework for understanding these techniques, which we term sampling adapters. Sampling adapters often lead to qualitatively better text, which raises the question: From a formal perspective, how are they changing the (sub)word-level distributions of language generation models? And why do these local changes lead to higher-quality text? We argue that the shift they enforce can be viewed as a trade-off between precision and recall: while the model loses its ability to produce certain strings, its precision rate on desirable text increases. While this trade-off is not reflected in standard metrics of distribution quality (such as perplexity), we find that several precision-emphasizing measures indeed indicate that sampling adapters can lead to probability distributions more aligned with the true distribution. Further, these measures correlate with higher sequence-level quality scores, specifically, MAUVE. https://github.com/rycolab/ sampling-adapters 1 Introduction The vast majority of natural language generation systems takes a probabilistic approach. The backbone of such an approach is a probability distribution over strings pθ for a specific target domain. While modern language models have achieved remarkable performance on standard measures of distribution quality, e.g., perplexity (Brown et al., 2020; Chowdhery et al., 2022; Hoffmann et al., 2022; OpenAI, 2023), they often fall short when applied out of the box for language generation tasks—both sampling directly from them and searching for the maximum-probability string under them can lead to dull, incoherent, and degenerate text (Holtzman et al., 2020; Eikema and Aziz, 2020; Welleck et al., 2020). Surprisingly, applying a post-hoc modification to pθ(· | y<t) often serves to dramatically improve the quality of the generated text (Nadeem et al., 2020; Pillutla et al., 2021; Wiher et al., 2022; Hewitt et al., 2022; Li et al., 2022). In this paper, we give a name to these methods, dubbing them sampling adapters. A sampling adapter can be formally defined as a simplex-to-simplex map α: ∆|V|−1 → ∆|V|−1 that systematically modifies the conditional distribution of a language model pθ(· | y<t), thus creating another language model α(pθ(· | y<t)) with a desired set of characteristics, e.g., it may only give non-zero probability to items assigned high probability under the original distribution. Sampling adapters often require little to no fine-tuning and can be implemented in just a few lines of code. Presumably due to their simplicity, sampling adapters have become a default tool in text generation pipelines, serving as the …

Content truncated to 500 words