Prefix Embeddings for In-context Machine Translation

Prefix Embeddings for In-context Machine Translation Suzanna Sia ssia1@jhu.edu Kevin Duh kevinduh@cs.jhu.edu Johns Hopkins University Abstract The class of large generative pretrained (GPT) language models have demonstrated the ability to translate with in-context examples, a phenomena known as few-shot prompting. However, they have not achieved state-of-art results for translating out of English. In this work, we in- vestigate an extremely lightweight fixed-parameter method for conditioning a large language model to better translate into the target language. Our method introduces additional embed- dings, refered to as prefix embeddings which do not interfere with the existing weights of the model. Using unsupervised and weakly supervised methods that train only 0.0001% of the model parameters, the simple method improves up to around 5 BLEU points over the base- line when a single prompt example is provided, and up to around 2 BLEU points when 20 prompt examples are provided across 3 domains and 3 languages. We analyze the resulting embeddings’ training dynamics, where they lie in the embedding space, and show that these conditional prefixes can be used for both in-context translation and diverse generation of the monolingual target sentence. 1 Introduction Under the paradigm of in-context learning,1 large language models have been shown to generate translations when provided with several priming examples, each of which consists of a source sentence and the translated target sentence. These examples, also known as “prompts”, are prefixed to the test source sentence, which then conditions the model to generate the test target sentence. Table 1 shows an example of this format, where [S1] and [S2] are separator tokens prefixing source and target sentence respectively. This prompt-and-translate phenomena, or in-context translation, presents itself as a new paradigm for Machine Translation applications. First, the ability to adapt to different task spec- ifications using prompts suggest that the same model can be used in multiple settings and do- mains. While there have been several multilingual translation models (Fan et al., 2021; Xue et al., 2021; Ma et al., 2021), the ability to perform unrelated tasks such as Question-Answering in addition to Translation is relatively new. This also presents an interesting shift from super- vised Neural Machine Translation (NMT) in terms of data requirements. These models are trained on massive amounts of web text which are not explicitly parallel.2 In contrast, mod- ern NMT models are trained with millions of lines of parallel text. Unsuprisingly, the lack of supervision comes at a cost. Translating out of English for in-context models still lags behind state-of-art possibly due to low data quality and/or disproportionate amounts of English. 1This has also been termed ‘few-shot prompting’ (Brown et al., 2020), but the field is increasingly converging on ‘in-context learning’ (Bommasani et al., 2021). 2This does not preclude the possibility that parallel sentences may exist in various forms in the crawled web text. Proceedings of the 15th Biennial Conference of the Association for Machine Translation in the Americas Orlando, USA, September 12-16, 2022. Volume 1: Research Track 45 [S1] So at this point, music diverged …

Content truncated to 500 words