Investigating Multi-source Active Learning for Natural Language Inference

Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics, pages 2187–2209 May 2-6, 2023 ©2023 Association for Computational Linguistics Investigating Multi-source Active Learning for Natural Language Inference Ard Snijders1 Douwe Kiela2 Katerina Margatina3 1University of Amsterdam 2Stanford University 3University of Sheffield ardsnijders@gmail.com Abstract In recent years, active learning has been suc- cessfully applied to an array of NLP tasks. However, prior work often assumes that train- ing and test data are drawn from the same dis- tribution. This is problematic, as in real-life settings data may stem from several sources of varying relevance and quality. We show that four popular active learning schemes fail to out- perform random selection when applied to unla- belled pools comprised of multiple data sources on the task of natural language inference. We reveal that uncertainty-based strategies perform poorly due to the acquisition of collective out- liers, i.e., hard-to-learn instances that hamper learning and generalization. When outliers are removed, strategies are found to recover and outperform random baselines. In further anal- ysis, we find that collective outliers vary in form between sources, and show that hard-to- learn data is not always categorically harmful. Lastly, we leverage dataset cartography to in- troduce difficulty-stratified testing and find that different strategies are affected differently by example learnability and difficulty. 1 Introduction In recent years, active learning (AL) (Cohn et al., 1996) has emerged as a promising avenue for data- efficient supervised learning (Zhang et al., 2022). AL has been successfully applied to a variety of NLP tasks, such as text classification (Zhang et al., 2016; Siddhant and Lipton, 2018; Prabhu et al., 2019; Ein-Dor et al., 2020; Margatina et al., 2022), entity recognition (Shen et al., 2017; Siddhant and Lipton, 2018; Lowell et al., 2019), part-of-speech tagging (Chaudhary et al., 2021) and neural ma- chine translation (Peris and Casacuberta, 2018; Liu et al., 2018; Zhao et al., 2020). However, these works share a major limitation: they often implicitly assume that unlabelled train- ing data comes from a single source1 (Houlsby 1Throughout the paper, we use the term “source” to de- Figure 1: The pool of unlabelled data consists of a source S, or multiple sources � i Si, for the single- source and multi-source AL setting, respectively. The in- domain test set follows the same distribution of sources. et al., 2011; Sener and Savarese, 2018; Huang et al., 2016; Gissin and Shalev-Shwartz, 2019; Margatina et al., 2021). We refer to this setting as single- source AL. The single-source assumption is prob- lematic for various reasons (Kirsch et al., 2021). In real-life settings we have no guarantees that all unlabelled data at our disposal will necessar- ily stem from the same distribution, nor will we have assurances that all examples are of consis- tent quality, or that they bear sufficient relevancy to our task. For instance, quality issues may arise when unlabelled data is collected through noisy processes with limited room for monitoring indi- vidual samples, such as web-crawling (Kreutzer et al., 2022). …

Content truncated to 500 words