Symbolic Knowledge Distillation: from General Language Models to Commonsense Models

Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 4602 - 4625 July 10-15, 2022 ©2022 Association for Computational Linguistics Symbolic Knowledge Distillation: from General Language Models to Commonsense Models Peter West†‡* Chandra Bhagavatula‡ Jack Hessel‡ Jena D. Hwang‡ Liwei Jiang†‡ Ronan Le Bras‡ Ximing Lu†‡ Sean Welleck†‡ Yejin Choi †‡* †Paul G. Allen School of Computer Science & Engineering, University of Washington ‡Allen Institute for Artificial Intelligence Abstract The common practice for training common- sense models has gone from–human–to–corpus– to–machine: humans author commonsense knowledge graphs in order to train common- sense models. In this work, we investigate an alternative, from–machine–to–corpus–to– machine: general language models author these commonsense knowledge graphs to train com- monsense models. Our study leads to a new framework, Sym- bolic Knowledge Distillation. As with prior art in Knowledge Distillation (Hinton et al., 2015), our approach uses larger models to teach smaller models. A key difference is that we distill knowledge symbolically–as text–in ad- dition to the resulting neural model. We distill only one aspect–the commonsense of a general language model teacher, allowing the student to be a different type of model, a common- sense model. Altogether, we show that careful prompt engineering and a separately trained critic model allow us to selectively distill high- quality causal commonsense from GPT-3, a general language model. Empirical results demonstrate that, for the first time, a human-authored commonsense knowl- edge graph is surpassed by our automatically distilled variant in all three criteria: quantity, quality, and diversity. In addition, it results in a neural commonsense model that surpasses the teacher model’s commonsense capabilities despite its 100x smaller size. We apply this to the ATOMIC resource, and will share our new symbolic knowledge graph and commonsense models1. 1 Introduction Prior works have suggested that pre-trained lan- guage models possess limited understanding of commonsense knowledge (Merrill et al., 2021; Tal- mor et al., 2021; Davis and Marcus, 2017) despite 1We will share this following the anonymity period. We have permission from OpenAI to release GPT-3 generations GPT-3 175B Parameters General Model ! ATOMIC10X 6.5M Examples Commonsense KG COMETdistil 1.5B Parameters Commonsense Model Symbolic Knowledge Distillation ! CRITIC Fine-tuned RoBERTa filters for quality Figure 1: Symbolic knowledge distillation extracts the commonsense from the large, general language model GPT-3, into 2 forms: a large commonsense knowledge graph ATOMIC10x, and a compact commonsense model COMETDIS TIL. The quality of this knowledge can be con- trolled and improved by adding a critic model, making GPT-3 a stronger teacher. otherwise stellar performance on leaderboards. As a result, symbolic commonsense knowledge graphs (Speer et al., 2017; Sap et al., 2019; Hwang et al., 2021) and corresponding neural representations (Bosselut et al., 2019; Hwang et al., 2021; Zhang et al., 2020b) have supplemented past models with commonsense capabilities. This has enabled di- verse downstream applications, including interac- tive learning through a conversational interface (Arabshahi et al., 2021), persona- and affect-aware conversation models (Kearns et al., 2020), figura- tive language …

Content truncated to 500 words