Ruler: Data Programming by Demonstration for Document Labeling

Findings of the Association for Computational Linguistics: EMNLP 2020, pages 1996–2005 November 16 - 20, 2020. c⃝2020 Association for Computational Linguistics 1996 Ruler: Data Programming by Demonstration for Document Labeling Sara Evensen Megagon Labs Mountain View, CA, USA sara@megagon.ai Chang Ge University of Waterloo Waterloo, ON, Canada c4ge@uwaterloo.ca C¸ a˘gatay Demiralp Megagon Labs Mountain View, CA, USA cagatay@megagon.ai Abstract Data programming aims to reduce the cost of curating training data by encoding domain knowledge as labeling functions over source data. As such it not only requires domain expertise but also programming experience, a skill that many subject matter experts lack. Additionally, generating functions by enumerating rules is not only time consuming but also inherently difficult, even for peo- ple with programming experience. In this paper we introduce RULER, an interactive system that synthesizes labeling rules using span-level interactive demonstrations over document examples. RULER is a first-of-a- kind implementation of data programming by demonstration (DPBD). This new framework aims to relieve users from the burden of writing labeling functions, enabling them to focus on higher-level semantic analysis, such as iden- tifying relevant signals for the labeling task. We compare RULER with conventional data programming through a user study conducted with 10 data scientists who were asked to create labeling functions for sentiment and spam classification tasks. Results show RULER is easier to learn and to use, and that it offers higher overall user-satisfaction while provid- ing model performances comparable to those achieved by conventional data programming. 1 Introduction Machine learning (ML) models used today are predominantly supervised and rely on large datasets labeled for training. However, the cost of collecting and maintaining labeled training data remains a bottleneck for training high-capacity supervised models [33]. Weak supervision methods such as crowdsourc- ing [15], distant supervision [26], and user-defined heuristics [10] enable the use of noisy or imprecise sources to gather large training datasets. Data Interactive Statistics Labeling Interaction Function Generation Figure 1: RULER enables the user to interactively generate a diverse set of labeling functions through simple, non-programmatic text annotations. Dynami- cally updated statistics allow the user to quickly test and evaluate ideas. programming [6, 30, 31] aims to address the difficulty of collecting labeled data by using a programmatic approach to weak supervision by heuristics, where domain experts are expected to provide data programs (labeling functions) incorpo- rating their domain knowledge. Prior work on data programming focuses on modeling and aggregating labeling functions written manually [30, 31] or generated automatically [13, 35] to create training data. However, little is known about user experience in writing labeling functions and how to improve it [2]. Many domain experts or lay users have little or no programming literacy. Even for proficient programmers, it is often difficult to convert domain knowledge to a set of rules by writing programs. We introduce RULER (Figure 1), an interactive system that enables more accessible data pro- gramming to create labeled training datasets for 1997 document classification models. RULER automat- ically generates labeling rules from user’s labeling rationales or …

Content truncated to 500 words