CLUES: A Benchmark for Learning Classifiers using Natural Language Explanations

Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics Volume 1: Long Papers, pages 6523 - 6546 May 22-27, 2022 c⃝2022 Association for Computational Linguistics CLUES: A Benchmark for Learning Classifiers using Natural Language Explanations Rakesh R Menon∗ Sayan Ghosh∗ Shashank Srivastava UNC Chapel Hill {rrmenon, sayghosh, ssrivastava}@cs.unc.edu Abstract Supervised learning has traditionally focused on inductive learning by observing labeled ex- amples of a task. In contrast, humans have the ability to learn new concepts from language. Here, we explore learning zero-shot classifiers for structured data1 purely from language from natural language explanations as supervision. For this, we introduce CLUES, a benchmark for Classifier Learning Using natural language ExplanationS, consisting of a range of classi- fication tasks over structured data along with natural language supervision in the form of ex- planations. CLUES consists of 36 real-world and 144 synthetic classification tasks. It con- tains crowdsourced explanations describing real-world tasks from multiple teachers and programmatically generated explanations for the synthetic tasks. We also introduce ExEnt, an entailment-based method for training clas- sifiers from language explanations, which ex- plicitly models the influence of individual ex- planations in making a prediction. ExEnt gen- eralizes up to 18% better (relative) on novel tasks than a baseline that does not use expla- nations. We identify key challenges in learn- ing from explanations, addressing which can lead to progress on CLUES in the future. Our code and datasets are available at: https: //clues-benchmark.github.io. 1 Introduction Humans have a remarkable ability to learn concepts through language (Chopra et al., 2019; Tomasello, 1999). For example, we can learn about poisonous mushrooms through an explanation like ‘a mush- room is poisonous if it has pungent odor’. Such ∗Equal contribution 1By structured data, we refer to data that can be reasonably represented using tables. This is a highly flexible format for representing a lot of real-world data (e.g., spreadsheets, traditional classification datasets in CSV format, single-table databases, as well as structured text-rich data such as emails), with a large variety in possible table schemas. �������������������������������� ����������������� ������������������������������ ���������������������� ��� ������������������������ ��� ����������������������������� ������������������������������� ���� ������ �������� ���������� ���������� ��������� ������� ������ ����� ������� Figure 1: We explore learning classification tasks over structured data from natural language supervision in form of explanations. The explanations provide declar- ative supervision about the task, and are not example- specific. This is an example from the UCI Mushroom dataset, one of the 36 real-world datasets for which we collect multiple sets of explanations in CLUES. an approach profoundly contrasts with the pre- dominant paradigm of machine learning, where algorithms extract patterns by looking at scores of labeled examples of poisonous and edible mush- rooms. However, it is unnatural to presume the availability of labeled examples for the heavy tail of naturally occurring concepts in the world. This work studies how models trained to learn from natural language explanations can generalize to novel tasks without access to labeled examples. While prior works in this area (Srivastava et al., 2017, 2018; Hancock et al., …

Content truncated to 500 words