FRMT: A Benchmark for Few-Shot Region-Aware Machine Translation

FRMT: A Benchmark for Few-Shot Region-Aware Machine Translation Parker Riley∗, Timothy Dozat∗, Jan A. Botha∗, Xavier Garcia∗, Dan Garrette, Jason Riesa, Orhan Firat, Noah Constant Google Research, USA {prkriley,tdozat,jabot,xgarcia,dhgarrette, riesa,orhanf,nconstant}@google.com Abstract We present FRMT, a new dataset and evalu- ation benchmark for Few-shot Region-aware Machine Translation, a type of style-targeted translation. The dataset consists of professional translations from English into two regional variants each of Portuguese and Mandarin Chinese. Source documents are selected to enable detailed analysis of phenomena of in- terest, including lexically distinct terms and distractor terms. We explore automatic evalua- tion metrics for FRMT and validate their cor- relation with expert human evaluation across both region-matched and mismatched rating scenarios. Finally, we present a number of baseline models for this task, and offer guide- lines for how researchers can train, evaluate, and compare their own models. Our dataset and evaluation code are publicly available: https://bit.ly/frmt-task. 1 Introduction Machine translation (MT) has made rapid ad- vances in recent years, achieving impressive performance for many language pairs, especially those with high amounts of parallel data available. Although the MT task is typically specified at the coarse level of a language (e.g., Spanish or Hindi), some prior work has explored finer-grained dis- tinctions, such as between regional varieties of Arabic (Zbib et al., 2012), or specific levels of politeness in German (Sennrich et al., 2016). Unfortunately, most approaches to style-targeted translation thus far rely on large, labeled train- ing corpora (Zbib et al., 2012; Lakew et al., 2018; Costa-juss`a et al., 2018; Honnet et al., 2018; Sajjad et al., 2020; Wan et al., 2020; Kumar et al., 2021), and in many cases these resources are unavail- able or expensive to create. ∗Equal contribution. We explore a setting for MT where unla- beled training data is plentiful for the desired language pair, but only a few parallel examples (0–100, called ‘‘exemplars’’) are annotated for the target varieties. As a specific use-case, we ex- amine translation into regional varieties: Brazilian vs. European Portuguese and Mainland vs. Tai- wan Mandarin. While these varieties are mutually intelligible, they often exhibit lexical, syntactic, or orthographic differences that can negatively im- pact an MT user’s experience. Figure 1 illustrates the use of exemplars to control the regional variety at inference time. MT systems that do not support region or style distinctions may be biased toward varieties with more available data (the ‘‘web-majority’’ vari- eties). We observe this bias in a widely used proprietary MT system, with measurable nega- tive effects for speakers of web-minority varieties (§6.2). One barrier to further research on this is- sue is the lack of a high-quality evaluation bench- mark. Thus, to encourage more access to language technologies for speakers of web-minority varie- ties and more equitable NLP research, we make the following contributions: (1) We construct and release FRMT, a new dataset for evaluating few-shot region-aware translation from English to Brazilian/European Portuguese and Mainland/ Taiwan Mandarin. (2) We evaluate predictions from a number of existing and custom-trained baseline systems on the …

Content truncated to 500 words