KnowMAN: Weakly Supervised Multinomial Adversarial Networks

Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 9549–9557 November 7–11, 2021. c⃝2021 Association for Computational Linguistics 9549 KnowMAN: Weakly Supervised Multinomial Adversarial Networks Luisa März ⋄,†, Ehsaneddin Asgari †, Fabienne Braune †, Franziska Zimmermann† and Benjamin Roth ⋄ ⋄ Digital Philology, Research Group Data Mining and Machine Learning, University of Vienna, Austria † NLP Expert Center, Data:Lab, Volkswagen AG, Munich, Germany Abstract The absence of labeled data for training neu- ral models is often addressed by leveraging knowledge about the specific task, resulting in heuristic but noisy labels. The knowledge is captured in labeling functions, which detect certain regularities or patterns in the training samples and annotate corresponding labels for training. This process of weakly supervised training may result in an over-reliance on the signals captured by the labeling functions and hinder models to exploit other signals or to generalize well. We propose KnowMAN, an adversarial scheme that enables to control in- fluence of signals associated with specific la- beling functions. KnowMAN forces the net- work to learn representations that are invari- ant to those signals and to pick up other sig- nals that are more generally associated with an output label. KnowMAN strongly improves results compared to direct weakly supervised learning with a pre-trained transformer lan- guage model and a feature-based baseline. 1 Introduction Neural approaches rely on labeled data sets for training. For many tasks and languages, such data is either scarce or not available at all. Knowledge- based weak supervision tackles this problem by employing labeling functions (LFs). LFs are manu- ally specified properties, e.g. keywords, that trigger the automatic annotation of a specific label. How- ever, these annotations contain noise and biases that need to be handled. A recent approach for denoising weakly super- vised data is Snorkel (Ratner et al., 2020). Snorkel focuses on estimating the reliability of LFs and of the resulting heuristic labels. However, Snorkel does not address biases on the input side of weakly supervised data, which might lead to learned repre- sentations that overfit the characteristics of specific LFs, hindering generalization. We address the prob- lem of overfitting to the LFs in this paper. Other approaches tackle such overfitting by deleting the LF signal completely from the input side of an annotated sample: For example, Go et al. (2009) strip out emoticons that were used for label- ing the sentiment in tweets, and Alt et al. (2019) mask the entities used for distant supervision of re- lation extraction training data (Mintz et al., 2009). However, as LFs are often constructed from the most prototypical and reliable signals (e.g., key- words), deleting them entirely from the feature space might – while preventing over-reliance on them – hurt prediction quality considerably. How- ever, we find a way to blur the signals of the LFs instead of removing them. In this work we propose KnowMAN (Knowledge-based Weakly Supervised Multi- nomial Adversarial Networks), a method for controllable soft deletion of LF signals, allowing a trade-off between reliance and generalization. …

Content truncated to 500 words