DaLC: Domain Adaptation Learning Curve Prediction for Neural Machine Translation

Findings of the Association for Computational Linguistics: ACL 2022, pages 1789 - 1807 May 22-27, 2022 c⃝2022 Association for Computational Linguistics DaLC: Domain Adaptation Learning Curve Prediction for Neural Machine Translation Cheonbok Park†, Hantae Kim†, Ioan Calapodescu‡ Hyunchang Cho†, and Vassilina Nikoulina‡,∗ †Papago, NAVER Corp.,‡NAVER LABS Europe {cbok.park,hantae.kim,hyunchang.cho}@navercorp.com {ioan.calapodescu, vassilina.nikoulina}@naverlabs.com Abstract Domain Adaptation (DA) of Neural Machine Translation (NMT) model often relies on a pre- trained general NMT model which is adapted to the new domain on a sample of in-domain parallel data. Without parallel data, there is no way to estimate the potential benefit of DA, nor the amount of parallel samples it would require. It is however a desirable func- tionality that could help MT practitioners to make an informed decision before investing resources in dataset creation. We propose a Domain adaptation Learning Curve prediction (DaLC) model that predicts prospective DA performance based on in-domain monolingual samples in the source language. Our model re- lies on the NMT encoder representations com- bined with various instance and corpus-level features. We demonstrate that instance-level is better able to distinguish between different do- mains compared to corpus-level frameworks proposed in previous studies (Xia et al., 2020; Kolachina et al., 2012). Finally, we perform in- depth analyses of the results highlighting the limitations of our approach, and provide direc- tions for future research. 1 Introduction The classical Domain Adaptation scenario (Freitag and Al-Onaizan, 2016; Luong and Manning, 2015) usually relies on an existing NMT model trained on large datasets originated from various sources. This general model is adapted to the new domain, with a small sample of in-domain parallel data, through finetuning or other DA methods. Without any parallel in-domain data, we cannot estimate the quality of general NMT model on the domain of interest, nor can we anticipate what will be the ben- efits of the DA and how much parallel in-domain data is required. In this work, we address the problem that trans- lation service providers may regularly face when ∗ Corresponding author receiving a request for a new domain translation. In such case, a new domain is often defined by its source language samples, and the translation provider needs to invest into in-domain parallel dataset creation in order to be able to perform evaluation and Domain Adaptation of its general model. Current state of the art research in NMT Domain Adaptation rarely provides any insights on the amount of data required to perform Domain Adaptation depending on input domain characteris- tics. This is however a desirable feature that would allow to (i) estimate the data creation cost (time and money-wise) for the client requesting an adaptation to completely new domain; (ii) make an informed decision on how to distribute fixed data creation budget when there is a need to handle multiple DA simultaneously (as we demonstrate further shar- ing this budget equally across all domains may not be optimal). The goal of this work is to gain better insights on Domain Adaptation dynamics, and provide practical guidelines for such …

Content truncated to 500 words