WALNUT: A Benchmark on Semi-weakly Supervised Learning for Natural Language Understanding

Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 873 - 899 July 10-15, 2022 ©2022 Association for Computational Linguistics WALNUT: A Benchmark on Semi-weakly Supervised Learning for Natural Language Understanding Guoqing Zheng Microsoft Research zheng@microsoft.com Giannis Karamanolakis Columbia University gkaraman@cs.columbia.edu Kai Shu Illinois Institute of Technology kshu@iit.edu Ahmed Hassan Awadallah Microsoft Research hassanam@microsoft.com Abstract Building machine learning models for natu- ral language understanding (NLU) tasks relies heavily on labeled data. Weak supervision has been proven valuable when large amount of labeled data is unavailable or expensive to ob- tain. Existing works studying weak supervi- sion for NLU either mostly focus on a specific task or simulate weak supervision signals from ground-truth labels. It is thus hard to compare different approaches and evaluate the benefit of weak supervision without access to a uni- fied and systematic benchmark with diverse tasks and real-world weak labeling rules. In this paper, we propose such a benchmark, named WALNUT1, to advocate and facilitate research on weak supervision for NLU. WALNUT con- sists of NLU tasks with different types, includ- ing document-level and token-level prediction tasks. WALNUT is the first semi-weakly super- vised learning benchmark for NLU, where each task contains weak labels generated by multiple real-world weak sources, together with a small set of clean labels. We conduct baseline evalu- ations on WALNUT to systematically evaluate the effectiveness of various weak supervision methods and model architectures. Our results demonstrate the benefit of weak supervision for low-resource NLU tasks and highlight interest- ing patterns across tasks. We expect WALNUT to stimulate further research on methodologies to leverage weak supervision more effectively. The benchmark and code for baselines are avail- able at aka.ms/walnut_benchmark. 1 Introduction To tackle natural language understanding (NLU) tasks via supervised learning, high-quality labeled examples are crucial. Recent advances on large pre-trained language models (Peters et al., 2018; Devlin et al., 2018; Radford et al., 2019) lead to impressive gains on NLU benchmarks, including 1WALNUT: Semi-WeAkly supervised Learning for Natural language Understanding Testbed Weak Rules Aggregated Label rule 1 rule 2 ... rule 50 Text 1 1 0 ... 1 1 Text 2 0 -1 ... -1 0 ... ... ... ... ... ... Text 1000 1 1 ... 1 1 Labels Text 1 1 Text 2 0 Text 3 1 WALNUT Semi-weakly Supervised Learning Few Labeled Data Weakly Labeled Data ● 8 NLU tasks with weak rules ○ Doc-level classification ○ Token-level classification ● 5 clean/weak splits per task Figure 1: WALNUT, a benchmark with 8 NLU tasks with real-world weak labeling rules. Each task in WALNUT comes with few labeled data and weakly labeled data for semi-weakly supervised learning. GLUE (Wang et al., 2018) and SuperGLUE (Wang et al., 2019), at the assumption that large amount of labeled examples are available. For many real- world applications, however, it is expensive and time-consuming to manually obtain large-scale high-quality labels, while it is relatively easier to obtain auxiliary supervision signals, or weak …

Content truncated to 500 words