Fully Hyperbolic Neural Networks

Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics Volume 1: Long Papers, pages 501 - 516 May 22-27, 2022 c⃝2022 Association for Computational Linguistics FewNLU: Benchmarking State-of-the-Art Methods for Few-Shot Natural Language Understanding Yanan Zheng∗12, Jing Zhou∗1, Yujie Qian3, Ming Ding1, Chonghua Liao1 Jian Li1, Ruslan Salakhutdinov4, Jie Tang†12, Sebastian Ruder† 5, Zhilin Yang†16 1Tsinghua University, 2BAAI, 3MIT CSAIL, 4Carnegie Mellon University, 5Google Research, 6Shanghai Qi Zhi Institute {zyanan, jietang, zhiliny}@tsinghua.edu.cn, zhouj18@mails.tsinghua.edu.cn, ruder@google.com Abstract The few-shot natural language understanding (NLU) task has attracted much recent attention. However, prior methods have been evaluated under a disparate set of protocols, which hin- ders fair comparison and measuring progress of the field. To address this issue, we intro- duce an evaluation framework that improves previous evaluation procedures in three key aspects, i.e., test performance, dev-test corre- lation, and stability. Under this new evalua- tion framework, we re-evaluate several state- of-the-art few-shot methods for NLU tasks. Our framework reveals new insights: (1) both the absolute performance and relative gap of the methods were not accurately estimated in prior literature; (2) no single method domi- nates most tasks with consistent performance; (3) improvements of some methods diminish with a larger pretrained model; and (4) gains from different methods are often complemen- tary and the best combined model performs close to a strong fully-supervised baseline. We open-source our toolkit, FewNLU, that imple- ments our evaluation framework along with a number of state-of-the-art methods. 1 2 1 Introduction Few-shot learning for natural language understand- ing (NLU) has been significantly advanced by pretrained language models (PLMs; Brown et al., 2020; Schick and Schütze, 2021a,b). With the goal of learning a new task with very few (usually less than a hundred) samples, few-shot learning benefits from the prior knowledge stored in PLMs. Various few-shot methods based on PLMs and prompting have been proposed (Liu et al., 2021b; Menon et al., 2021; Gao et al., 2020). ∗The authors have contributed equally to this work. † Corresponding Authors. 1Leaderboard: https://fewnlu.github.io 2Code available at https://github.com/THUDM/ FewNLU Although the research of few-shot NLU is devel- oping rapidly, the lack of a standard evaluation protocol has become an obstacle hindering fair comparison between various methods on a com- mon ground and measuring progress of the field. While some works (Schick and Schütze, 2021b; Menon et al., 2021) experimented with a fixed set of hyper-parameters, prior work (Perez et al., 2021; Zhang et al., 2020) noted that such a setting might be exposed to the risk of overestimation .3 Other works (Liu et al., 2021b; Gao et al., 2020; Perez et al., 2021) proposed to use a small development set to select hyper-parameters, but their evaluation protocols vary in a few key aspects (e.g., how to construct data splits), which in fact lead to large differences as we will show (in Section 4.2). The above phenomena highlight the need for a com- mon protocol for the evaluation of few-shot NLU methods. However, the fact that few-shot learn- ing is extremely sensitive …

Content truncated to 500 words