Specializing Multi-domain NMT via Penalizing Low Mutual Information

Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 10015 - 10026 December 7-11, 2022 ©2022 Association for Computational Linguistics Specializing Multi-domain NMT via Penalizing Low Mutual Information Jiyoung Lee†∗, Hantae Kim‡, Hyunchang Cho‡ Edward Choi†, and Cheonbok Park‡ †KAIST , ‡Papago, NAVER Corp. {jiyounglee0523, edwardchoi}@kaist.ac.kr {hantae.kim,hyunchang.cho,cbok.park}@navercorp.com Abstract Multi-domain Neural Machine Translation (NMT) trains a single model with multiple do- mains. It is appealing because of its efficacy in handling multiple domains within one model. An ideal multi-domain NMT should learn dis- tinctive domain characteristics simultaneously, however, grasping the domain peculiarity is a non-trivial task. In this paper, we investi- gate domain-specific information through the lens of mutual information (MI) and propose a new objective that penalizes low MI to become higher. Our method achieved the state-of-the- art performance among the current competitive multi-domain NMT models. Also, we empiri- cally show our objective promotes low MI to be higher resulting in domain-specialized multi- domain NMT. 1 Introduction Multi-domain Neural Machine Translation (NMT) (Sajjad et al., 2017; Farajian et al., 2017) has been an attractive topic due to its efficacy in handling multiple domains with a single model. Ideally, a multi-domain NMT should capture both gen- eral knowledge (e.g., sentence structure, common words) and domain-specific knowledge (e.g., do- main terminology) unique in each domain. While the shared knowledge can be easily acquired via sharing parameters across domains (Kobus et al., 2017), obtaining domain specialized knowledge is a challenging task. Haddow and Koehn (2012) demonstrate that a model trained on multiple do- mains sometimes underperforms the one trained on a single domain. Pham et al. (2021) shows that separate domain-specific adaptation modules are not sufficient to fully-gain specialized knowledge. In this paper, we reinterpret domain specialized knowledge from mutual information (MI) perspec- tive and propose a method to strengthen it. Given ∗ Work done during an internship at NAVER Corp. MI(𝑫; 𝒀|𝑿) 𝟎 calculation computing totals 𝑨 𝑩 Source Beschreib … Summenberechnung für ein gegebenes Feld oder einen gegebenen Ausdruck. Reference Describes a way of computing totals for a given field or expression. A (Baseline) Describes the kind of calculation for a given field or expression. B (Ours) Describes the way of computing totals for a given field or expression . Figure 1: Overview of two models with different MI distributions. The example sentence is from IT domain. Model A mostly has low MI and Model B has large MI. For an identical sample, model A outputs a generic term ‘calculation’ while model B properly maintains ‘computing totals’. a source sentence X, target sentence Y , and cor- responding domain D, the MI between D and the translation Y |X (i.e., MI(D; Y |X)) measures the dependency between the domain and the trans- lated sentence. Here, we assume that the larger MI(D; Y |X), the more the translation incorporates domain knowledge. Low MI is undesirable because it indicates the model is not sufficiently utilizing do- main characteristics in translation. In other words, low MI can be interpreted as …

Content truncated to 500 words