Data-to-text Generation for Severely Under-Resourced Languages with GPT-3.5: A Bit of Help Needed from Google Translate (WebNLG 2023)

Proceedings of the Workshop on Multimodal, Multilingual Natural Language Generation and Multilingual WebNLG Challenge (MM-NLG 2023), pages 80–86 September 12, 2023 ©2023 Association for Computational Linguistics Data-to-text Generation for Severely Under-Resourced Languages with GPT-3.5: A Bit of Help Needed from Google Translate Michela Lorandi and Anya Belz ADAPT Research Centre Dublin City University, Ireland {michela.lorandi,anya.belz}@adaptcentre.ie Abstract LLMs like GPT are great at tasks involving En- glish which dominates in their training data. In this paper, we look at how they cope with tasks involving languages that are severely under- represented in their training data, in the con- text of data-to-text generation for Irish, Mal- tese, Welsh and Breton. During the prompt- engineering phase we tested a range of prompt types and formats on GPT-3.5 and 4 with a small sample of example input/output pairs. We then fully evaluated the two most promis- ing prompts in two scenarios: (i) direct genera- tion into the under-resourced language, and (ii) generation into English followed by translation into the under-resourced language. We find that few-shot prompting works better for direct generation into under-resourced languages, but that the difference disappears when pivoting via English. The few-shot + translation system variants were submitted to the WebNLG 2023 shared task where they outperformed competi- tor systems by substantial margins in all lan- guages on all metrics. We conclude that good performance on under-resourced languages can be achieved out-of-the box with state-of-the-art LLMs. However, our best results (for Welsh) remain well below the lowest ranked English system at WebNLG’20. 1 Introduction Pretrained large language models (LLMs) are ev- erywhere, achieving state-of-the-art-results across many application areas. LLMs display strong out- of-the-box abilities at a wide range of different tasks far beyond language-based tasks. However, these abilities have been explored mostly for En- glish which dominates text online, hence the data on which LLMs are trained. In the work reported here, we explore to what extent state-of-the-art LLMs can help with tasks involving severely under- resourced languages which by definition would have been very scarce in their training data. More specifically, we address the challenging task of data-to-text generation for four severely under-resourced languages: Irish, Maltese, Welsh and Breton. Developing any new system is difficult in a severely under-resourced scenario, more so where the task has language-specific input/output combinations as in data-to-text and text-to-text. The paper is structured as follows. Section 2 describes data and task, Section 3 presents the gen- eral approach, prompt engineering, and the specific systems we fully evaluated. Experimental set-up and results are outlined in Section 4, and Section 5 provides discussion and conclusions. All code and results are available on Github: https://github.com/DCU-NLG/DCU-NLG-PBN. 2 Data and Task The WebNLG’23 (?) data consists of 1,778 dev items for each language, 1,399 test items for Bre- ton, and 1,665 for Welsh, Irish and Maltese, that were manually translated by professional trans- lators from the English originals. Additionally 13,211 training items are provided where texts were automatically translated from English. As in all WebNLG shared tasks, WebNLG 2023 systems …

Content truncated to 500 words