Probable Human Knowledge Elicitation for Efficient Human-in-the-Loop Text Classification without Any Label-Related Data

Probable Human Knowledge Elicitation for Efficient Human-in-the-Loop Text Classification without Any Label-Related Data Osamu Saisho NTT / Tokyo, Japan osamu.saisho.vm@hco.ntt.co.jp Abstract This paper presents a new human-in-the- loop text classification framework with no labeled data incorporating weak supervision and Bayesian active learning with a seman- tic clustering constraint. Labeling function (LF)-based weak supervision is a promising method to alleviate the huge labeling cost in supervised learning. However, most previ- ous studies relied on the impractical assump- tion of sufficient and well-designed pre-built LFs, even though they also require a heavy workload to implement. The proposed frame- work is the first intended to substantially re- duce this workload by a human-in-the-loop approach from scratch. Specifically, it inter- actively suggests data samples to help humans implement an LF considering the actual hu- man workload. Its superiority is achieved by Bayesian active learning with a semantic clus- tering constraint for not only enhancing label- ing utility but also reducing human workload. The results of the user studies with 16 vol- unteers demonstrate that the proposed method can achieve higher performance with fewer LFs and help humans implement more accu- rate LFs in a shorter work time than conven- tional methods. 1 Introduction Supervised learning assuming the availability of massive labeled data has been put into practical use, but this assumption has also been the main obsta- cle to expanding its use. To overcome this obstacle, weak supervision (WS) has gained much attention as a promising method. It is incomplete, inexact, or inaccurate supervision as an alternative to strong supervision such as a fully labeled dataset (Zhou, 2017). Specifically, labeling function (LF) is a rep- resentative approach used in both academia and in- dustry (Bach et al., 2019; Fries et al., 2019). LF is a human-defined function based on heuristics, col- lective knowledge, or implicit rules for indirect la- beling instead of manual direct labeling. Its main strengths are its high performance and robustness to task changes (Ratner et al., 2016; Bach et al., 2017). Even though LF-based WS alleviates the huge and redundant labeling cost, it usually depends on suf- ficient and well-designed pre-built rules and meta- rules for LFs. When implementing LFs in the wild from scratch, the difference in the required work- load and the potential benefits on each LF must be considered. Thus, there are two issues left to be solved simultaneously for the practical LF-based WS framework. First, only high-performance LFs should be implemented. Second, humans should be able to implement more accurate LFs with a lighter workload, that is, in a shorter work time. The most appropriate and pragmatic approach to solve both issues is a human-in-the-loop approach. Especially, active learning (AL) (Settles, 2009) is a key method to solve the first issue. A few re- cent studies have attempted to develop an AL-like framework for WS (Wang et al., 2019; Saisho et al., 2021). These methods try implementing effective LFs interactively, but they disregard the workload for each LF. In addition, their effectiveness has been verified only …

Content truncated to 500 words