Translation Transformers Rediscover Inherent Data Domains

Proceedings of the Sixth Conference on Machine Translation (WMT), pages 599–613 November 10–11, 2021. ©2021 Association for Computational Linguistics 599 Translation Transformers Rediscover Inherent Data Domains Maksym Del∗, Elizaveta Korotkova∗, Mark Fishel Institute of Computer Science University of Tartu, Estonia {maksim,lisa_k,mark}@tartunlp.ai Abstract Many works proposed methods to improve the performance of Neural Machine Transla- tion (NMT) models in a domain/multi-domain adaptation scenario. However, an understand- ing of how NMT baselines represent text domain information internally is still lack- ing. Here we analyze the sentence repre- sentations learned by NMT Transformers and show that these explicitly include the infor- mation on text domains, even after only see- ing the input sentences without domains la- bels. Furthermore, we show that this inter- nal information is enough to cluster sentences by their underlying domains without supervi- sion. We show that NMT models produce clus- ters better aligned to the actual domains com- pared to pre-trained language models (LMs). Notably, when computed on document-level, NMT cluster-to-domain correspondence nears 100%. We use these findings together with an approach to NMT domain adaptation using au- tomatically extracted domains. Whereas previ- ous work relied on external LMs for text clus- tering, we propose re-using the NMT model as a source of unsupervised clusters. We per- form an extensive experimental study compar- ing two approaches across two data scenar- ios, three language pairs, and both sentence- level and document-level clustering, showing equal or significantly superior performance compared to LMs. 1 Introduction Neural machine translation (NMT, Bahdanau et al., 2015; Vaswani et al., 2017b) heavily depends on training data and the text domains covered in it. Full-scale NMT Transfomer models (Vaswani et al., 2017b) are usually trained on a mix of corpora from several domains (Barrault et al., 2020). However, the field lacks an understanding of how these NMT models represent the training data domains in their inner vector spaces. ∗Equal contribution This paper consists of two main parts. First, we analyze domain representations learned by the NMT Transformer. We consider sentence-level as well as document-level representations via mean pooling of token contextual embeddings. Our anal- ysis shows that NMT models explicitly learn to include the domain information in their representa- tional spaces across layers. Furthermore, we show that text representations preserve enough domain- specific information to reveal the underlying do- mains with Principal Component Analysis and k- means clustering without supervision. In the case of document-level clustering, the result of k-means matches the original corpora almost perfectly. In the case of sentence-level clustering, we observe some deviation between automatic clusters and the original corpora that the sentences belong to, show- ing corpus heterogeneity on the sentence level. Aharoni and Goldberg (2020) previously re- vealed that a similar property exists in pre-trained language models (LMs). We compare LMs with NMT Transformers in how well we can extract un- supervised domain clusters from them and show the superiority of NMT models. In the second part of the paper, we show how to effectively utilize our analysis to improve an ex- …

Content truncated to 500 words