Embedding Space Correlation as a Measure of Domain Similarity

Proceedings of the 12th Conference on Language Resources and Evaluation (LREC 2020), pages 2431–2439 Marseille, 11–16 May 2020 c⃝ European Language Resources Association (ELRA), licensed under CC-BY-NC 2431 Embedding Space Correlation as a Measure of Domain Similarity Anne Beyer1, G¨oran Kauermann2, Hinrich Sch¨utze1 1Center for Information and Language Processing (CIS), 2Department of Statistics Ludwig-Maximilians-Universit¨at Munich, Germany anne.beyer@campus.lmu.de, goeran.kauermann@stat.lmu.de, inquiries@cislmu.org Abstract Prior work has determined domain similarity using text-based features of a corpus. However, when using pre-trained word embeddings, the underlying text corpus might not be accessible anymore. Therefore, we propose the CCA measure, a new measure of domain similarity based directly on the dimension-wise correlations between corresponding embedding spaces. Our results suggest that an inherent notion of domain can be captured this way, as we are able to reproduce our findings for different domain comparisons for English, German, Spanish and Czech as well as in cross-lingual comparisons. We further find a threshold at which the CCA measure indicates that two corpora come from the same domain in a monolingual setting by applying permutation tests. By evaluating the usability of the CCA measure in a domain adaptation application, we also show that it can be used to determine which corpora are more similar to each other in a cross-domain sentiment detection task. Keywords: domain similarity, word embeddings, domain adaptation 1. Introduction The application of neural network approaches to solve Nat- ural Language Processing (NLP) tasks has led to the de- velopment of language representations that capture intrin- sic linguistic information without the need for task specific, manual feature extraction by experts. This is achieved by so called word embeddings, which are high dimensional vec- tor representations that can be induced from unlabeled data. Despite the success of embeddings, there are still open re- search questions about what exactly these representations capture. In this paper, we address the question as how dif- ferences in domains and languages are reflected by embed- ding spaces. Previous work has shown that it is possible to align embedding spaces across domains and languages (e.g. Mikolov et al. (2013a), Artetxe et al. (2016), Barnes et al. (2018)), but to the best of our knowledge, no task inde- pendent, multi-lingual analysis of the underlying structural similarities has been carried out so far. A systematic notion of similarity is also interesting from an application point of view. A widely used technique is to augment a model’s capabilities by using pre-trained word embeddings, based on larger or more diverse corpora. This can be applied across different text domains or languages. In both cases, it can be useful to know how similar available pre-trained embeddings are to the target corpus in order to pick the most similar or to estimate the amount of fine tun- ing or adaptation necessary. While there exist approaches to determine corpus similarity based on text features (usu- ally word frequency distributions or language model per- plexity), these resources are often not available when using pre-trained word embeddings. Our contribution is therefore to propose the CCA measure, …

Content truncated to 500 words