Unsupervised Expressive Rules Provide Explainability and Assist Human Experts Grasping New Domains

Findings of the Association for Computational Linguistics: EMNLP 2020, pages 2678–2697 November 16 - 20, 2020. c⃝2020 Association for Computational Linguistics 2678 Unsupervised Expressive Rules Provide Explainability and Assist Human Experts Grasping New Domains Eyal Shnarch∗, Leshem Choshen∗, Guy Moshkowich, Noam Slonim, Ranit Aharonov IBM Research {eyals, leshem.choshen, noams}@il.ibm.com, {guy.moshkowich, ranit.aharonov2}@ibm.com Abstract Approaching new data can be quite deterrent; you do not know how your categories of in- terest are realized in it, commonly, there is no labeled data at hand, and the performance of domain adaptation methods is unsatisfactory. Aiming to assist domain experts in their first steps into a new task over a new corpus, we present an unsupervised approach to reveal complex rules which cluster the unexplored corpus by its prominent categories (or facets). These rules are human-readable, thus provid- ing an important ingredient which has become in short supply lately - explainability. Each rule provides an explanation for the common- ality of all the texts it clusters together. We present an extensive evaluation of the use- fulness of these rules in identifying target cat- egories, as well as a user study which assesses their interpretability. 1 Introduction A common scenario faced by subject matter experts tackling a new text understanding task is getting to know a new dataset, for which there is no la- beled data. Understanding the unexplored data, and collecting first insights from it, is always a slow process. Often, the expert is trying to categorize the data, and potentially build a system to auto- matically identify these categories. For example, an expert may be looking at customer complaints, aiming to understand their types or categories, and then building a system that will categorize com- plaints. Or she may be analyzing contracts, aiming to identify the types of legal commitments. In other cases, the expert may be trying to iden- tify a certain class of texts, and this class may be composed of unknown underlying sub-types or cat- egories. Consider a data scientist looking for all ∗First two authors equally contributed to this work. arguments, related to a suggested policy, raised in a public participation forum. These arguments may be of several types, which are a-priori unknown. When facing a new task, with no labeled data, but with domain expertise, a practical first step is to manually write rules that identify some texts from a certain category the expert is aware of and aiming to identify (e.g., a certain complaint type). With these seed examples, experts can better understand the occurrences of the target category in the new corpus, and use them as the initial set of labeled ex- amples, towards the goal of having enough labeled data to facilitate supervised learning. However, oftentimes, the categories underlying the data are not known a-priori, and may be a part of what the expert aims to identify (e.g., what are the types of complaints). Since new data may mean new underlying categories, domain adaptation is not always applicable, and often results in unsatis- fying performance (Ziser …

Content truncated to 500 words