Searching for Needles in a Haystack: On the Role of Incidental Bilingualism in PaLM’s Translation Capability

Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics Volume 1: Long Papers, pages 9432–9452 July 9-14, 2023 ©2023 Association for Computational Linguistics Searching for Needles in a Haystack: On the Role of Incidental Bilingualism in PaLM’s Translation Capability Eleftheria Briakou ebriakou@cs.umd.edu Colin Cherry colincherry@google.com George Foster fosterg@google.com Abstract Large, multilingual language models exhibit surprisingly good zero- or few-shot machine translation capabilities, despite having never seen the intentionally-included translation ex- amples provided to typical neural translation systems. We investigate the role of inciden- tal bilingualism—the unintentional consump- tion of bilingual signals, including translation examples—in explaining the translation capa- bilities of large language models, taking the Pathways Language Model (PaLM) as a case study. We introduce a mixed-method approach to measure and understand incidental bilin- gualism at scale. We show that PaLM is exposed to over 30 million translation pairs across at least 44 languages. Furthermore, the amount of incidental bilingual content is highly correlated with the amount of monolin- gual in-language content for non-English lan- guages. We relate incidental bilingual con- tent to zero-shot prompts and show that it can be used to mine new prompts to improve PaLM’s out-of-English zero-shot translation quality. Finally, in a series of small-scale abla- tions, we show that its presence has a substan- tial impact on translation capabilities, although this impact diminishes with model scale. 1 Introduction Recent work has shown that large language mod- els (LLMs) exhibit impressive capabilities in per- forming various natural language generation tasks, even in the zero-shot paradigm. In particular, such models have shown interesting machine translation (MT) capabilities (Brown et al., 2020; Chowdhery et al., 2022; Vilar et al., 2022)—especially when translating into English, despite never having been explicitly and intentionally exposed to translation data in the way their supervised counterparts are. This raises the question: where do these translation capabilities come from? We hypothesize that the translation capabilities of LLMs connect to incidental bilingualism: the unintentional consumption of bilingual text within a single training instance. To test this hypothesis, we take PaLM (Chowdhery et al., 2022)—a 540- billion parameter Transformer language model—as a case study. We first conduct a large-scale analy- sis of its training data in order to characterize the nature and quantity of bilingual text, then perform experiments to assess the impact of this text on translation performance. To measure incidental bilingualism at scale, we develop a processing pipeline that alternates be- tween quantitative and qualitative analysis (§3): first detect bilingual versus monolingual text us- ing a language tagger, then qualitatively analyze the nature of bilingual text, and finally measure the amount of translation data within bilingual instances. Our analysis spans 44 languages, for which we study bilingualism paired with English. Our findings are: • In all, 1.4% of PALM’s training instances are detected as bilingual, while 0.34% contain at least one translated sentence pair. We were able to mine such pairs across all languages studied; therefore, none of these languages is truly zero-shot in the context of translation. • The number …

Content truncated to 500 words