In-context Learning as Maintaining Coherency: A Study of On-the-fly Machine Translation Using Large Language Models

Proceedings of Machine Translation Summit XIX, Vol. 1: Research Track, pages 173–185 September 4–8, 2023, Macau SAR, China. ©2023 The authors. This article is licensed under a Creative Commons Attribution 4.0 International License (CC BY 4.0). 173 In-context Learning as Maintaining Coherency: A Study of On-the-fly Machine Translation Using Large Language Models Suzanna Sia ssia1@jh.edu Kevin Duh kevinduh@cs.jhu.edu Johns Hopkins University Abstract The phenomena of in-context learning has typically been thought of as “learning from ex- amples”. In this work which focuses on Machine Translation, we present a perspective of in-context learning as the desired generation task maintaining coherency with its context, i.e., the prompt examples. We first investigate randomly sampled prompts across 4 domains, and find that translation performance improves when shown in-domain prompts. Next, we investi- gate coherency for the in-domain setting, which uses prompt examples from a moving window. We study this with respect to other factors that have previously been identified in the litera- ture such as length, surface similarity and sentence embedding similarity. Our results across 3 models (GPTNeo2.7B, Bloom3B, XGLM2.9B), and three translation directions (en→{pt, de, fr}) suggest that the long-term coherency of the prompts and the test sentence is a good indicator of downstream translation performance. In doing so, we demonstrate the efficacy of in-context Machine Translation for on-the-fly adaptation. Code for this paper is available at https://github.com/suzyahyah/icl_coherence_mt. 1 Introduction The in-context learning paradigm describes a phenomena where large autoregressive language models perform a task when shown examples (known as prompts) in the prefix (Brown et al., 2020; Bommasani et al., 2021). In-context Machine Translation is a relatively new paradigm that uses large autoregressive Language Models to carry out the task of Machine Translation (MT) by being shown translation pairs in the prefix. From a practitioner’s viewpoint, in-context learning presents itself as an attractive approach for rapidly adapting a translation model on-the- fly. Previous strategies for adapting a pre-trained MT model still require additional engineering or training of the model, e.g fine-tuning with in-domain data using adaptor layers (Philip et al., 2020). Instead, simply changing the inputs to the model might be an effective way to adapt on-the-fly without any model modification. Previous work assumes that the role of the prompt context is to allow the model to “learn by examples”. This has led to formulating the task of prompt selection as selecting examples that are similar to the source sentence being translated. Semantic similarity based on sentence embeddings (Liu et al., 2021) and BM25 have been proposed to select examples to present as “demonstrations” (Rubin et al., 2021). This approach was further expanded by Agrawal et al. (2022) who use a heuristic version optimizing for word coverage. 174 Translate English to French. English: A discomfort which lasts .. French: Un malaise qui dure English: HTML is a language for formatting French: HTML est un langage de formatage ... ... English: After you become comfortable with formatting .. French: Table 1: A single continuous input sequence presented to the model for decoding a …

Content truncated to 500 words