Programmable Annotation with Diversed Heuristics and Data Denoising

Proceedings of the 29th International Conference on Computational Linguistics, pages 2681–2691 October 12–17, 2022. 2681 Programmable Annotation with Diversed Heuristics and Data Denoising †Ernie Chang, *Alex Marin, †Vera Demberg †Dept. of Language Science and Technology, Saarland University *Microsoft Corporation, Redmond, WA cychang@coli.uni-saarland.de Abstract Neural natural language generation (NLG) and understanding (NLU) models are costly and re- quire massive amounts of annotated data to be competitive. Recent data programming frame- works address this bottleneck by allowing hu- man supervision to be provided as a set of la- beling functions to construct generative mod- els that synthesize weak labels at scale. How- ever, these labeling functions are difficult to build from scratch for NLG/NLU models, as they often require complex rule sets to be spec- ified. To this end, we propose a novel data pro- gramming framework that can jointly construct labeled data for language generation and un- derstanding tasks – by allowing the annotators to modify an automatically-inferred alignment rule set between sequence labels and text, in- stead of writing rules from scratch. Further, to mitigate the effect of poor quality labels, we propose a dually-regularized denoising mecha- nism for optimizing the NLU and NLG models. On two benchmarks we show that the frame- work can generate high-quality data that comes within a 1.48 BLEU and 6.42 slot F1 of the 100% human-labeled data (42k instances) with just 100 labeled data samples – outperforming benchmark annotation frameworks and other semi-supervised approaches. 1 Introduction Modern machine learning systems require large amounts of labeled data. For many applications, such labeled data is created by getting humans to explicitly label each training example. However, the standard labeling process that involves Wizard- of-Oz (Kelley, 1984) and other crowd-sourcing ap- proaches (e.g. (Wen et al., 2017; Coucke et al., 2018; Budzianowski et al., 2018)) is restricted to the level of individual examples, and so are slow and static (Ratner et al., 2019). As a result, they are not only costly but require relabeling for any fine-grained domain revisions. Manual Annotation x x x x x x x x x x x Distant Annotation MR: restaurant_name=Ristorante food=Italian price_range=high Text: Ristorante is an expensive Italian restaurant . Labeling Function Figure 1: Annotation scenario: Each × represents a labeled data instance. The annotation framework allows to generalize from few human-labeled instances (inner) to large amounts of weakly labeled data (outer) by building rules (alignments) between sequence labels and text via labeling functions. Data programming is a successful paradigm where humans provide low cost labeling rules writ- ten in programming languages to build imperfect training sets, which are then denoised for further improvements (Ratner et al., 2016a, 2017). How- ever, two caveats exist: (1) Heuristic rules are costly to construct from scratch as exhaustive align- ments between text and sequence labels have to be specified manually (Evensen et al., 2020), and espe- cially strenuous for language generation that trans- forms meaning representations (MR) or slots into textual descriptions (e.g. (Reiter and Dale, 2000; Barzilay and Lapata, 2005)) where NLG is …

Content truncated to 500 words