TGEA: An Error-Annotated Dataset and Benchmark Tasks for TextGeneration from Pretrained Language Models

Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing, pages 550–564 August 1–6, 2021. ©2021 Association for Computational Linguistics 550 Structural Knowledge Distillation: Tractably Distilling Information for Structured Predictor Xinyu Wang⋄‡♠, Yong Jiang†∗♠, Zhaohui Yan⋄♠, Zixia Jia⋄♠, Nguyen Bach†, Tao Wang†, Zhongqiang Huang†, Fei Huang†, Kewei Tu⋄∗ ⋄School of Information Science and Technology, ShanghaiTech University Shanghai Engineering Research Center of Intelligent Vision and Imaging Shanghai Institute of Microsystem and Information Technology, Chinese Academy of Sciences University of Chinese Academy of Sciences †DAMO Academy, Alibaba Group {wangxy1,jiazx,yanzhh,tukw}@shanghaitech.edu.cn {yongjiang.jy,nguyen.bach}@alibaba-inc.com {leeo.wangt,z.huang,f.huang}@alibaba-inc.com Abstract Knowledge distillation is a critical technique to transfer knowledge between models, typi- cally from a large model (the teacher) to a more fine-grained one (the student). The objec- tive function of knowledge distillation is typ- ically the cross-entropy between the teacher and the student’s output distributions. How- ever, for structured prediction problems, the output space is exponential in size; there- fore, the cross-entropy objective becomes in- tractable to compute and optimize directly. In this paper, we derive a factorized form of the knowledge distillation objective for structured prediction, which is tractable for many typical choices of the teacher and student models. In particular, we show the tractability and empir- ical effectiveness of structural knowledge dis- tillation between sequence labeling and depen- dency parsing models under four different sce- narios: 1) the teacher and student share the same factorization form of the output struc- ture scoring function; 2) the student factoriza- tion produces more fine-grained substructures than the teacher factorization; 3) the teacher factorization produces more fine-grained sub- structures than the student factorization; 4) the factorization forms from the teacher and the student are incompatible.1 1 Introduction Deeper and larger neural networks have led to sig- nificant improvement in accuracy in various tasks, but they are also more computationally expensive and unfit for resource-constrained scenarios such ∗Yong Jiang and Kewei Tu are the corresponding authors. ♠: Equal contributions. ‡: This work was conducted when Xinyu Wang was interning at Alibaba DAMO Academy. 1Our code is publicly available at https://github. com/Alibaba-NLP/StructuralKD. as online serving. An interesting and viable solu- tion to this problem is knowledge distillation (KD) (Buciluˇa et al., 2006; Ba and Caruana, 2014; Hin- ton et al., 2015), which can be used to transfer the knowledge of a large model (the teacher) to a smaller model (the student). In the field of natu- ral language processing (NLP), for example, KD has been successfully applied to compress massive pretrained language models such as BERT (Devlin et al., 2019) and XLM-R (Conneau et al., 2020) into much smaller and faster models without sig- nificant loss in accuracy (Tang et al., 2019; Sanh et al., 2019; Tsai et al., 2019; Mukherjee and Has- san Awadallah, 2020). A typical approach to KD is letting the stu- dent mimic the teacher model’s output probabil- ity distributions on the training data by using the cross-entropy objective. For structured prediction problems, however, the output space …

Content truncated to 500 words