Additive Interventions Yield Robust Multi-Domain Machine Translation Models

Proceedings of the Seventh Conference on Machine Translation (WMT), pages 220 - 232 December 7-8, 2022 ©2022 Association for Computational Linguistics Additive Interventions Yield Robust Multi-Domain Machine Translation Models Elijah Rippeth∗ Department of Computer Science University of Maryland erip@cs.umd.edu Matt Post Microsoft mattpost@microsoft.com Abstract Additive interventions are a recently-proposed mechanism for controlling target-side attributes in neural machine translation. In contrast to tag-based approaches which manipulate the raw source sequence, interventions work by directly modulating the encoder representa- tion of all tokens in the sequence. We ex- amine the role of additive interventions in a large-scale multi-domain machine translation setting and compare its performance in vari- ous inference scenarios. We find that while the performance difference is small between intervention-based systems and tag-based sys- tems when the domain label matches the test do- main, intervention-based systems are robust to label error, making them an attractive choice un- der label uncertainty. Further, we find that the superiority of single-domain fine-tuning comes under question when training data size is scaled, contradicting previous findings. 1 Introduction Multi-domain machine translation (MDMT) is the paradigm in which a single model is trained to ser- vice many domains by training on multiple corpora covering disparate labeled domains. The goal of MDMT is not only to provide high quality general machine translation enabled by knowledge trans- fer across domains, but also to enable high quality domain-specific machine translation when a model is provided cues about the target domain, used to control the generation. Though an intuitive task, the expectations surrounding the task were only recently formalized by Pham et al. (2021) in which the authors provided both a set of functional re- quirements demanded of successful MDMT mod- els and an experimental framework under which those requirements can be tested. Pham et al. (2021) explored several mechanisms for controlling domain, ranging from simple tag- ∗ Work was done during an internship at Microsoft based approaches to meta-learning based mecha- nisms. According to the functional requirements outlined by the authors, no method meets all the ex- pectations demanded of effective multi-domain ma- chine translators, though the experiments were run on a relatively small dataset of only in-domain data. The primary remaining expectations, according to the authors, are the superiority of fine-tuning based methods as compared to these methods which can control the target domain, and the ability to accom- modate fuzzy or uncertain domains. This framework is useful, but the authors leave open several other questions regarding the state of MDMT. The first of these is data size. Previous experiments focused only on relatively small, in- domain data in an otherwise high-resource setting of English-French and found that most models pale in comparison to models fine-tuned on a single domain. We wonder whether this fine-tuning su- periority conclusion holds under a more realistic paradigm in which models trained on large, out-of- domain datasets are fine-tuned on in-domain data. While pretraining and fine-tuning on in-domain data can yield strong in-domain performance—as observed by the authors—this is likely to be at …

Content truncated to 500 words