Program Transfer for Answering Complex Questions over Knowledge Bases

Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics Volume 1: Long Papers, pages 745 - 758 May 22-27, 2022 c⃝2022 Association for Computational Linguistics PRBOOST: Prompt-Based Rule Discovery and Boosting for Interactive Weakly-Supervised Learning Rongzhi Zhang Georgia Tech rongzhi.zhang@gatech.edu Yue Yu Georgia Tech yueyu@gatech.edu Pranav Shetty Georgia Tech pranav.shetty@gatech.edu Le Song MBZUAI le.song@mbzuai.ac.ae Chao Zhang Georgia Tech chaozhang@gatech.edu Abstract Weakly-supervised learning (WSL) has shown promising results in addressing label scarcity on many NLP tasks, but manually designing a comprehensive, high-quality labeling rule set is tedious and difficult. We study interac- tive weakly-supervised learning—the problem of iteratively and automatically discovering novel labeling rules from data to improve the WSL model. Our proposed model, named PR- BOOST, achieves this goal via iterative prompt- based rule discovery and model boosting. It uses boosting to identify large-error instances and then discovers candidate rules from them by prompting pre-trained LMs with rule tem- plates. The candidate rules are judged by hu- man experts, and the accepted rules are used to generate complementary weak labels and strengthen the current model. Experiments on four tasks show PRBOOST outperforms state-of-the-art WSL baselines up to 7.1%, and bridges the gaps with fully supervised mod- els.Our Implementation is available at https: //github.com/rz-zhang/PRBoost. 1 Introduction Weakly-supervised learning (WSL) has recently at- tracted increasing attention to mitigate the label scarcity issue in many NLP tasks. In WSL, the training data are generated by weak labeling rules obtained from sources such as knowledge bases, frequent patterns, or human experts. The weak la- beling rules can be matched with unlabeled data to create large-scale weak labels, allowing for train- ing NLP models with much lower annotation cost. WSL has recently achieved promising results in many tasks including text classification (Awasthi et al., 2020; Mekala and Shang, 2020; Meng et al., 2020; Yu et al., 2021b), relation extraction (Zhou et al., 2020), and sequence tagging (Lison et al., 2020; Safranchik et al., 2020; Li et al., 2021b). Despite its success, WSL is limited by two ma- jor factors: 1) the labeling rules, and 2) the static learning process. First, it is challenging to provide a comprehensive and high-quality set of labeling rules a priori. Labeling rules are often human- written (Ratner et al., 2017; Hancock et al., 2018), but the process of writing labeling rules is tedious and time-consuming even for experts. A few works attempt to automatically discover labeling rules by mining labeled data (Varma and Ré, 2018), or enumerating predefined types. However, the pre- extracted rules are restricted to frequent patterns or predefined types, which are inadequate for training an accurate model. Second, most existing WSL methods are static and can suffer from the noise in the initial weak supervision (Ratner et al., 2017; Zhou et al., 2020; Yu et al., 2021b; Meng et al., 2020; Zhang et al., 2022). As the labeling rule set remains fixed during model training, the ini- tial errors can be amplified, resulting in an over- fitted end model. Interactive rule discovery has been …

Content truncated to 500 words