Investigating the Translation Performance of a Large Multilingual Language Model: the Case of BLOOM

Investigating the Translation Performance of a Large Multilingual Language Model: the Case of BLOOM Rachel Bawden Inria, Paris, France rachel.bawden@inria.fr Franc¸ois Yvon Universit´e Paris-Saclay, CNRS, LISN francois.yvon@cnrs.fr Abstract The NLP community recently saw the re- lease of a new large open-access mul- tilingual language model, BLOOM (Big- Science et al., 2022) covering 46 languages. We focus on BLOOM’s multilingual abil- ity by evaluating its machine translation performance across several datasets (WMT, Flores-101 and DiaBLa) and language pairs (high- and low-resourced). Our results show that 0-shot performance suffers from overgeneration and generating in the wrong language, but this is greatly improved in the few-shot setting, with very good results for a number of language pairs. We study several aspects including prompt design, model sizes, cross-lingual transfer and the use of discursive context. 1 Introduction Large language models (LLMs) trained at scale with simple objectives have been found to achieve results that match dedicated systems on numerous NLP tasks (Radford et al., 2019), as long as tasks are formulated as text generation though “prompt- ing” (Liu et al., 2023). LLMs’ multi-task perfor- mance can even be improved with “instruction” fine-tuning (Sanh et al., 2022; Muennighoff et al., 2022), few-shot priming, and better strategies to select or learn prompts (Petroni et al., 2019; Shin et al., 2020; Schick and Sch¨utze, 2021; Lester et al., 2021; Wei et al., 2022). In multilingual set- tings, their performance on machine translation (MT) tasks, as measured by automatic scores, is © 2023 The authors. This article is licensed under a Creative Commons 4.0 licence, no derivative works, attribution, CC- BY-ND. often close to state of the art, even when mostly trained on monolingual data (Brown et al., 2020). Moreover, prompting-based MT offers the prospect of better control of outputs, e.g. in terms of quality, style and dialect (Garcia and Firat, 2022). However, these abilities remain poorly understood, as LLM analyses primarily focus on their multitask rather than multilingual ability (see however (Vilar et al., 2022; Zhang et al., 2023; Moslem et al., 2023), which we discuss in Section 2). In this work, we focus on the MT performance of BLOOM (BigScience et al., 2022), a (family of) open-access multilingual LLM(s), designed and trained by the collaborative BigScience project.1 Our main aims are to (i) evaluate BLOOM’s zero- and multi-shot behaviour, (ii) study the effect of prompt design, (iii) evaluate a diverse set of lan- guage pairs and (iv) assess its ability to use linguis- tic context. Our main conclusions, which extend those in (BigScience et al., 2022), are (i) 0-shot ability is blighted by overgeneration and generating in the wrong language, (ii) using few-shot improves both issues, with results much closer to state of the art across datasets and language pairs, (iii) there are clear transfer effects, with high scores for languages not officially seen in training, and successful trans- fer across language pairs via few-shot examples and (iv) although linguistic context does not lead to higher scores, there is evidence that BLOOM’s translations are influenced by …

Content truncated to 500 words