Timeline Summarization based on Event Graph Compression via Time-Aware Optimal Transport

Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 650–663 November 7–11, 2021. c⃝2021 Association for Computational Linguistics 650 Active Learning by Acquiring Contrastive Examples Katerina Margatina† Giorgos Vernikos‡∗ Loïc Barrault† Nikolaos Aletras† †University of Sheffield ‡EPFL ∗HEIG-VD {k.margatina, l.barrault, n.aletras}@sheffield.ac.uk georgios.vernikos@epfl.ch Abstract Common acquisition functions for active learn- ing use either uncertainty or diversity sam- pling, aiming to select difficult and diverse data points from the pool of unlabeled data, re- spectively. In this work, leveraging the best of both worlds, we propose an acquisition function that opts for selecting contrastive ex- amples, i.e. data points that are similar in the model feature space and yet the model outputs maximally different predictive likeli- hoods. We compare our approach, CAL (Con- trastive Active Learning), with a diverse set of acquisition functions in four natural language understanding tasks and seven datasets. Our experiments show that CAL performs consis- tently better or equal than the best performing baseline across all tasks, on both in-domain and out-of-domain data. We also conduct an extensive ablation study of our method and we further analyze all actively acquired datasets showing that CAL achieves a better trade-off between uncertainty and diversity compared to other strategies. 1 Introduction Active learning (AL) is a machine learning paradigm for efficiently acquiring data for anno- tation from a (typically large) pool of unlabeled data (Lewis and Catlett, 1994; Cohn et al., 1996; Settles, 2009). Its goal is to concentrate the human labeling effort on the most informative data points that will benefit model performance the most and thus reducing data annotation cost. The most widely used approaches to acquiring data for AL are based on uncertainty and diversity, often described as the “two faces of AL” (Das- gupta, 2011). While uncertainty-based methods leverage the model predictive confidence to select difficult examples for annotation (Lewis and Gale, 1994; Cohn et al., 1996), diversity sampling ex- ploits heterogeneity in the feature space by typi- cally performing clustering (Brinker, 2003; Bodó Figure 1: Illustrative example of our proposed method CAL. The solid line (model decision boundary) sepa- rates data points from two different classes (blue and orange), the coloured data points represent the labeled data and the rest are the unlabeled data of the pool. et al., 2011). Still, both approaches have core limi- tations that may lead to acquiring redundant data points. Algorithms based on uncertainty may end up choosing uncertain yet uninformative repetitive data, while diversity-based methods may tend to se- lect diverse yet easy examples for the model (Roy and McCallum, 2001). The two approaches are orthogonal to each other, since uncertainty sam- pling is usually based on the model’s output, while diversity exploits information from the input (i.e. feature) space. Hybrid data acquisition functions that combine uncertainty and diversity sampling have also been proposed (Shen et al., 2004; Zhu et al., 2008; Ducoffe and Precioso, 2018; Ash et al., 2020; Yuan et al., 2020; Ru et al., 2020). In this work, we aim to leverage characteristics …

Content truncated to 500 words