Generic and Specialized Word Embeddings for Multi-Domain Machine Translation

Generic and Specialized Word Embeddings for Multi-Domain Machine Translation MinhQuang Pham†‡, Josep Crego†, Franc¸ois Yvon‡, Jean Senellart† †SYSTRAN / 5 rue Feydeau, 75002 Paris, France firstname.lastname@systrangroup.com ‡LIMSI, CNRS, Universit´e Paris-Saclay 91405 Orsay, France firstname.lastname@limsi.fr Abstract Supervised machine translation works well when the train and test data are sampled from the same distribution. When this is not the case, adaptation techniques help ensure that the knowledge learned from out-of-domain texts generalises to in-domain sentences. We study here a related setting, multi- domain adaptation, where the number of domains is poten- tially large and adapting separately to each domain would waste training resources. Our proposal transposes to neu- ral machine translation the feature expansion technique of (Daum´e III, 2007): it isolates domain-agnostic from domain- specific lexical representations, while sharing the most of the network across domains. Our experiments use two architec- tures and two language pairs: they show that our approach, while simple and computationally inexpensive, outperforms several strong baselines and delivers a multi-domain system that successfully translates texts from diverse sources. 1. Introduction Owing to the development of flexible and powerful architec- tures based on neural networks [1, 2, 3, 4], Machine Transla- tion (MT) has made significant progresses over the past years and constitute to date the standard for most production en- gines. The development of MT systems, be they neural or statistical, require very large parallel corpora consisting of millions of sentence pairs, a resource that only exist in very few application domains and language pairs. A lot of the re- cent research effort has thus focused on developing MT sys- tems in restricted data conditions, for instance building mul- tilingual MTs which enable zero-shot translation [5, 6, 7]. Another important scenario for industrial MT is to adapt a neural system trained using parallel data in one domain to the peculiarity of other domains. Domain Adaption (DA) in MT is an old issue [8, 9], which comes in various guises and for which a number of solutions have been studied. See the recent survey of [10] for neural MT. The typical setting is supervised adaptation, where a (small) amount of data in the target domain of inter- est is used to fine-tune the parameters of a system trained on a large amount of texts in a source domain. We study here a different scenario, multi-domain adaptation [11, 12], where we would like to use heterogenous data sources to train a unique system that would work well for all domains. This allows us to be both data efficient (all data is used to train all domains) and computationally efficient (we only train one system). Multi-domain adaptation is conceptually close to multilingual MT, or more generally to multi-task learning [13] and can be approached in a number of ways. We adapt here ideas of [14] to neural MT. Our main hy- pothesis is that domains mostly differ at the lexical level, due to cross-domain polysemy, which motivates domain specific embeddings. By contrast, the deeper layers, which arguably model more abstract linguistic …

Content truncated to 500 words