A Study on the Effectiveness of Large Language Models for Translation with Markup

Proceedings of Machine Translation Summit XIX, Vol. 1: Research Track, pages 148–159 September 4–8, 2023, Macau SAR, China. ©2023 The authors. This article is licensed under a Creative Commons Attribution 4.0 International License (CC BY 4.0). 148 A Study on the Effectiveness of Large Language Models for Translation with Markup Raj Dabre raj.dabre@nict.go.jp Hideki Tanaka hideki.tanaka@nict.go.jp National Institute of Information and Communications Technology, Japan Bianka Buschbeck bianka.buschbeck@sap.com Miriam Exel miriam.exel@sap.com SAP SE, Walldorf, Germany Abstract In this paper we evaluate the utility of large language models (LLMs) for translation of text with markup in which the most important and challenging aspect is to correctly transfer markup tags while ensuring that the content, both, inside and outside tags is correctly translated. While LLMs have been shown to be effective for plain text translation, their effectiveness for structured document translation is not well understood. To this end, we experiment with BLOOM and BLOOMZ, which are open-source multilingual LLMs, using zero, one and few- shot prompting, and compare with a domain-specific in-house NMT system using a detag- and-project approach for markup tags. We observe that LLMs with in-context learning exhibit poorer translation quality compared to the domain-specific NMT system, however, they are effective in transferring markup tags, especially the large BLOOM model (176 billion parame- ters). This is further confirmed by our human evaluation which also reveals the types of errors of the different tag transfer techniques. While LLM-based approaches come with the risk of losing, hallucinating and corrupting tags, they excel at placing them correctly in the translation. 1 Introduction Recent work involving Large Language Models (LLMs) has shown impressive performance in various Natural Language Processing (NLP) tasks. These models have the ability to per- form few-shot (or in-context) learning based on prompts, an alternative to fine-tuning, requiring only a forward pass of the neural network (Brown et al., 2020). Prompts are instructions in natural language given as input to LLMs along with a test sequence, allowing a few exam- ples (i.e. few-shot) to be fed to the model at test time. Researchers have shown that LLMs via prompting can be effective as Machine Translation (MT) systems (Brown et al., 2020; Wei et al., 2022; Chowdhery et al., 2022; Zhang et al., 2023; Hendy et al., 2023; Bawden and Yvon, 2023), whose quality approaches that of traditional encoder-decoder neural MT (NMT) systems trained or fine-tuned on parallel corpora. The majority of the aforementioned research has been conducted on plain text, neglecting the practical application of MT for text containing markup, see Table 1, where the challenge is to properly transfer markup tags within the translatable content from the source to the target language. Given that a significant portion of web-based content and proprietary or business documents requiring translation comes in structured for- 149 en Click <uicontrol>Prepayment</uicontrol>. ja <uicontrol>前払</uicontrol>をクリックします。 Table 1: Example with inline markup (in gray), taken from Buschbeck et al. (2022). mats like HTML pages or Microsoft Office files, it is important to understand the effectiveness of LLMs in handling this …

Content truncated to 500 words