Compositional Generalization in Dependency Parsing

Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics Volume 1: Long Papers, pages 583 - 595 May 22-27, 2022 c⃝2022 Association for Computational Linguistics Improving Meta-learning for Low-resource Text Classification and Generation via Memory Imitation Yingxiu Zhao1, Zhiliang Tian1∗, Huaxiu Yao2, Yinhe Zheng3, Dongkyu Lee1 Yiping Song4, Jian Sun3, Nevin L. Zhang1 1The Hong Kong University of Science and Technology, Hong Kong SAR, China 2Stanford University, 3Alibaba Group 4Department of Computer Science, Peking University, Beijing, China {yzhaocx,ztianac,dleear,lzhang}@cse.ust.hk,huaxiu@cs.stanford.edu {zhengyinhe.zyh,jian.sun}@alibaba-inc.com,songyiping@pku.edu.cn, Abstract Building models of natural language process- ing (NLP) is challenging in low-resource sce- narios where only limited data are available. Optimization-based meta-learning algorithms achieve promising results in low-resource sce- narios by adapting a well-generalized model initialization to handle new tasks. Nonetheless, these approaches suffer from the memoriza- tion overfitting issue, where the model tends to memorize the meta-training tasks while ignor- ing support sets when adapting to new tasks. To address this issue, we propose a memory imitation meta-learning (MemIML) method that enhances the model’s reliance on support sets for task adaptation. Specifically, we intro- duce a task-specific memory module to store support set information and construct an im- itation module to force query sets to imitate the behaviors of some representative support- set samples stored in the memory. A theo- retical analysis is provided to prove the effec- tiveness of our method, and empirical results also demonstrate that our method outperforms competitive baselines on both text classifica- tion and generation tasks. 1 Introduction Building natural language processing (NLP) mod- els in low-resource scenarios is of great importance in practical applications because labeled data are scarce. Meta-learning-based methods (Thrun and Pratt, 2012) have been commonly used in such scenarios owing to their fast adaptation ability. Notable successes have been achieved by meta- learning on low-resource NLP tasks, such as multi- domain sentiment classification (Yu et al., 2018; Geng et al., 2019) and personalized dialogue gen- eration (Madotto et al., 2019; Song et al., 2020; Zheng et al., 2020). Among different meta-learning approaches (Hospedales et al., 2021), optimization-based ap- ∗Corresponding author proaches have been widely used in various low- resource NLP scenarios (Madotto et al., 2019; Qian and Yu, 2019; Li et al., 2020; Mi et al., 2019) be- cause they are model-agnostic and easily applica- ble. Concretely, optimization-based meta-learning algorithms aim to learn a well-generalized global model initialization θ that can quickly adapt to new tasks within a few steps of gradient updates. In the meta-training process, we first train θ on a support set (i.e., a few training samples of a new task i) to obtain task-specific parameters θ′ i. Then, we opti- mize θ based on the performance of θ′ i on a query set (i.e., another set of samples in task i). Despite its effectiveness, optimization-based meta-learning algorithms usually suffer from the memorization overfitting issue 1 (Yin et al., 2020; Rajendran et al., 2020), where the learned model tends to solve all the meta-training tasks by memo- rization, rather than learning how to quickly …

Content truncated to 500 words