Domain Divergences: A Survey and Empirical Analysis

Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 1830–1849 June 6–11, 2021. ©2021 Association for Computational Linguistics 1830 Domain Divergences: A Survey and Empirical Analysis Abhinav Ramesh Kashyap†, Devamanyu Hazarika†, Min-Yen Kan†, Roger Zimmermann † †National University of Singapore, Singapore {abhinav, hazarika, kanmy, rogerz}@comp.nus.edu.sg Abstract Domain divergence plays a significant role in estimating the performance of a model in new domains. While there is a significant litera- ture on divergence measures, researchers find it hard to choose an appropriate divergence for a given NLP application. We address this shortcoming by both surveying the literature and through an empirical study. We develop a taxonomy of divergence measures consisting of three classes — Information-theoretic, Ge- ometric, and Higher-order measures and iden- tify the relationships between them. Further, to understand the common use-cases of these measures, we recognise three novel applica- tions – 1) Data Selection, 2) Learning Repre- sentation, and 3) Decisions in the Wild – and use it to organise our literature. From this, we identify that Information-theoretic measures are prevalent for 1) and 3), and Higher-order measures are more common for 2). To fur- ther help researchers choose appropriate mea- sures to predict drop in performance – an im- portant aspect of Decisions in the Wild, we perform correlation analysis spanning 130 do- main adaptation scenarios, 3 varied NLP tasks and 12 divergence measures identified from our survey. To calculate these divergences, we consider the current contextual word repre- sentations (CWR) and contrast with the older distributed representations. We find that tra- ditional measures over word distributions still serve as strong baselines, while higher-order measures with CWR are effective. 1 Introduction Standard machine learning models do not perform well when tested on data from a different target do- main. The performance in a target domain largely depends on the domain divergence (Ben-David et al., 2010) – a notion of distance between the two domains. Thus, efficiently measuring and reducing divergence is crucial for adapting models to the new domain — the topic of domain adaptation. Diver- gence also has practical applications in predicting the performance drop of a model when adapted to new domains (Van Asch and Daelemans, 2010), and in choosing among alternate models (Xia et al., 2020). Given its importance, researchers have invested much effort to define and measure domain diver- gence. Linguists use register variation to capture va- rieties in text – the difference between distributions of the prevalent features in two registers (Biber and Conrad, 2009). Other measures include probabilis- tic measures like H-divergence (Ben-David et al., 2010), information theoretic measures like Jenssen- Shannon and Kullback-Leibler divergence (Plank and van Noord, 2011; Van Asch and Daelemans, 2010) and measures using higher-order moments of random variables like Maximum Mean Dis- crepancy (MMD) and Central Moment Discrep- ancy (CMD) (Gretton et al., 2007; Zellinger et al., 2017). The proliferation of divergence measures challenges researchers in choosing an appropriate measure for a given application. To help guide …

Content truncated to 500 words