Distantly Supervised Course Concept Extraction in MOOCs with Academic Discipline

Consistency Regularization Training for Compositional Generalization Yongjing Yin1,2∗, Jiali Zeng3, Yafu Li1,2, Fandong Meng3, Jie Zhou3, Yue Zhang2,4† 1 Zhejiang University 2 School of Engineering, Westlake University 3 Pattern Recognition Center, WeChat AI, Tencent Inc 4 Institute of Advanced Technology, Westlake Institute for Advanced Study {yinyongjing,liyafu}@westlake.edu.cn {lemonzeng,fandongmeng,withtomzhou}@tencent.com yue.zhang@wias.org.cn Abstract Existing neural models have difficulty general- izing to unseen combinations of seen compo- nents. To achieve compositional generalization, models are required to consistently interpret (sub)expressions across contexts. Without mod- ifying model architectures, we improve the ca- pability of Transformer on compositional gen- eralization through consistency regularization training, which promotes representation con- sistency across samples and prediction consis- tency for a single sample. Experimental results on semantic parsing and machine translation benchmarks empirically demonstrate the effec- tiveness and generality of our method. In addi- tion, we find that the prediction consistency scores on in-distribution validation sets can be an alternative for evaluating models during training, when commonly-used metrics are not informative. 1 Introduction Compositional (systematic) generalization refers to the ability to understand and produce a potentially infinite number of novel combinations of known atoms (Chomsky, 2009; Janssen and Partee, 1997). Humans exhibit exceptional compositional gener- alization capability, easily producing and under- standing unseen linguistic expressions by recom- bining the learned rules (Montague and Thomason, 1975). Therefore, it is also regarded as a desired property for neural networks. Despite the impres- sive progress in language modeling (Vaswani et al., 2017; Liu et al., 2019; Raffel et al., 2020), the sequence-to-sequence (seq2seq) models have been demonstrated inefficient in capturing the composi- tional rules, thus failing to generalize to novel com- positions (Lake and Baroni, 2018; Keysers et al., 2020a; Kim and Linzen, 2020; Li et al., 2021). ∗This work was done as an intern at Pattern Recognition Center, WeChat AI, Tencent Inc, China. †Corresponding author Achieving compositional generalization requires a model to perform consistently in the interpre- tation assigned to a (sub)expression across con- texts (Janssen and Partee, 1997; Dankers et al., 2022). For example, the interpretation of a phrase “the book” is consistent whether it is described by a modifier “he likes”, in both semantic parsing and machine translation domains (Kim and Linzen, 2020; Li et al., 2021). To improve the consistency, most existing work considers a change of neural architecture to suit particular composition or gen- eralization test sets (Chen et al., 2020b; Guo et al., 2020b; Yin et al., 2022; Zheng and Lapata, 2022), which limits their potentials in real world applica- tions. Recently, the Transformer architecture has be- come the standard for natural language process- ing (NLP), particularly in supporting large pre- trained language models (PLMs) such as T5 and GPT-3 (Raffel et al., 2020; Brown et al., 2020). The Transformer-based PLMs have significantly improved few-shot fine-tuning and even made effi- cient zero-shot learning possible. As a result, there has been a trend towards developing data-centric AI (Koch et al., 2021; Jakubik et al., 2022), where the focus is on data preparation and training strategies rather than on the model …

Content truncated to 500 words