Foreseeing the Benefits of Incidental Supervision

Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 1782–1800 November 7–11, 2021. c⃝2021 Association for Computational Linguistics 1782 Foreseeing the Benefits of Incidental Supervision Hangfeng He† Mingyuan Zhang† Qiang Ning‡∗ Dan Roth† † University of Pennsylvania ‡ Amazon {hangfeng,myz,danroth}@seas.upenn.edu qning@amazon.com Abstract Real-world applications often require im- proved models by leveraging a range of cheap incidental supervision signals. These could in- clude partial labels, noisy labels, knowledge- based constraints, and cross-domain or cross- task annotations – all having statistical asso- ciations with gold annotations but not exactly the same. However, we currently lack a prin- cipled way to measure the benefits of these signals to a given target task, and the com- mon practice of evaluating these benefits is through exhaustive experiments with various models and hyperparameters. This paper stud- ies whether we can, in a single framework, quantify the benefits of various types of inci- dental signals for a given target task without going through combinatorial experiments. We propose a unified PAC-Bayesian motivated in- formativeness measure, PABI, that character- izes the uncertainty reduction provided by in- cidental supervision signals. We demonstrate PABI’s effectiveness by quantifying the value added by various types of incidental signals to sequence tagging tasks. Experiments on named entity recognition (NER) and question answering (QA) show that PABI’s predictions correlate well with learning performance, pro- viding a promising way to determine, ahead of learning, which supervision signals would be beneficial.1 1 Introduction The supervised learning paradigm, where direct supervision signals are available in high-quality and large amounts, has been struggling to fulfill needs in many real-world AI applications. As a result, researchers and practitioners often resort to datasets that are not collected directly for the tar- get task but capture some phenomena useful for it ∗Part of this work was done while the author was at Allen Institute for AI. 1Our code is publicly available at https://github. com/CogComp/PABI. 6 University Princeton . Alan studied Turing at I-ORG B-ORG O B-PER O I-PER O Words: Gold: I-ORG O B-PER O Partial: Noisy: Auxiliary: Constraints: I-ORG O O B-PER O O B-PER I B O B O I O I-ORG O O I-PER Figure 1: An example of NER with various inciden- tal supervision signals: partial labels (some missing labels in structured outputs), noisy labels (some incor- rect labels), auxiliary labels (labels of another task, e.g. named entity detection in the figure), and constraints in structured learning (e.g. the BIO constraint where I-X must follow B-X or I-X (Ramshaw and Marcus, 1999) in the figure). (Pan and Yang, 2009; Vapnik and Vashist, 2009; Roth, 2017; Kolesnikov et al., 2019). However, it remains unclear how to predict the benefits of these incidental signals on our target task beforehand. For example, given two incidental signals that are relevant to the target task, it is still difficult for us to predict which one is more beneficial. Therefore, the common practice is often trial-and-error: per- form experiments with different combinations of datasets and learning protocols, often exhaustively, to …

Content truncated to 500 words