Models Teaching Models: Improving Model Accuracy with Slingshot Learning

Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics, pages 3233–3247 May 2-6, 2023 ©2023 Association for Computational Linguistics Models Teaching Models: Improving Model Accuracy with Slingshot Learning Lachlan O’Neill, Nandini Anantharama, Satya Borgohain, Simon D. Angus SoDa Labs, Monash Business School, Monash University, Clayton Australia Abstract One significant obstacle to the successful appli- cation of machine learning to real-world data is that of labeling: it is often prohibitively ex- pensive to pay an ethical amount for the human labor required to label a dataset successfully. Human-in-the-loop techniques such as active learning can reduce the cost, but the required human time is still significant and many fixed costs remain. Another option is to employ pre- trained transformer models as labelers at scale, which can yield reasonable accuracy and sig- nificant cost savings. However, such models can still be expensive to use due to their high computational requirements, and the opaque nature of these models is not always suitable in applied social science and public use contexts. We propose a novel semi-supervised method, named Slingshot Learning, in which we itera- tively and selectively augment a small human- labeled dataset with labels from a high-quality "teacher" model to slingshot the performance of a "student" model in a cost-efficient manner. This reduces the accuracy trade-off required to use these simpler algorithms without disrupting their benefits, such as lower compute require- ments, better interpretability, and faster infer- ence. We define and discuss the slingshot learn- ing algorithm and demonstrate its effectiveness on several benchmark tasks, using ALBERT to teach a simple Naive Bayes binary classifier. We experimentally demonstrate that Slingshot learning effectively decreases the performance gap between the teacher and student models. We also analyze its performance in several sce- narios and compare different variants of the algorithm. 1 Introduction In standard computational linguistics modeling, the typical strategy for achieving high model accuracy is to obtain a large fraction of high-quality human labels on the dataset at hand. However, it is not un- common in applied social science settings to find that the scope of human labeling of a dataset is highly constrained (Liew et al., 2014). For exam- ple, it may be prohibitively expensive to pay an ethical amount for human labeling. Alternatively, the project’s institutional access to raw corpora may have ended, or perhaps the domain experts are no longer available. One solution is to use an active learning ap- proach where a model is trained concurrently with the labeling process to “maximize a model’s per- formance gain while annotating the fewest samples possible”(Ren et al., 2021). While this can be more efficient than unguided labeling, it still requires significant human labeling resources as the learn- ing process is fundamentally human-guided. Some issues are relatively fixed costs, such as bias, re- peatability, and initial training requirements; this enforces a non-trivial lower bound on the efficiency of human-in-the-loop active learning methods. Ac- tive learning also does not help if the data has al- ready been collected, and …

Content truncated to 500 words