SUPERB-SG: Enhanced Speech processing Universal PERformance Benchmark for Semantic and Generative Capabilities

Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics Volume 1: Long Papers, pages 8479 - 8492 May 22-27, 2022 c⃝2022 Association for Computational Linguistics SUPERB-SG: Enhanced Speech processing Universal PERformance Benchmark for Semantic and Generative Capabilities Hsiang-Sheng Tsai1˚, Heng-Jui Chang1˚, Wen-Chin Huang2˚, Zili Huang3˚, Kushal Lakhotia4˚, Shu-wen Yang1, Shuyan Dong4, Andy T. Liu1, Cheng-I Jeff Lai5, Jiatong Shi6, Xuankai Chang6, Phil Hall7, Hsuan-Jui Chen1, Shang-Wen Li4, Shinji Watanabe6, Abdelrahman Mohamed4, Hung-yi Lee1 1National Taiwan University, Taiwan 2Nagoya University, Japan 3Johns Hopkins University, USA 4Meta AI, USA 5Massachusetts Institute of Technology, USA 6Carnegie Mellon University, USA 7LXT {r09922024, b06901020}@ntu.edu.tw wen.chinhuang@g.sp.m.is.nagoya-u.ac.jp hzili1@jhu.edu, kushall@fb.com shangwel@fb.com, shinjiw@ieee.org abdo@fb.com, hungyilee@ntu.edu.tw Abstract Transfer learning has proven to be crucial in advancing the state of speech and natural lan- guage processing research in recent years. In speech, a model pre-trained by self-supervised learning transfers remarkably well on multi- ple tasks. However, the lack of a consistent evaluation methodology is limiting towards a holistic understanding of the efficacy of such models. SUPERB was a step towards intro- ducing a common benchmark to evaluate pre- trained models across various speech tasks. In this paper, we introduce SUPERB-SG, a new benchmark focused on evaluating the seman- tic and generative capabilities of pre-trained models by increasing task diversity and dif- ficulty over SUPERB. We use a lightweight methodology to test the robustness of repre- sentations learned by pre-trained models under shifts in data domain and quality across dif- ferent types of tasks. It entails freezing pre- trained model parameters, only using simple task-specific trainable heads. The goal is to be inclusive of all researchers, and encourage effi- cient use of computational resources. We also show that the task diversity of SUPERB-SG coupled with limited task supervision is an ef- fective recipe for evaluating the generalizabil- ity of model representation. ˚Equal contribution. 1 Introduction Transfer learning is a paradigm in machine learn- ing that has been very effective for natural lan- guage processing (NLP) (Peters et al., 2018; De- vlin et al., 2019; Liu et al., 2019; Lan et al., 2019; Dong et al., 2019; Yang et al., 2019; Raffel et al., 2020; Lewis et al., 2019; Conneau et al., 2020), and speech processing (van den Oord et al., 2018; Rivière et al., 2020; Chung et al., 2019; Schneider et al., 2019; Baevski et al., 2020b; Hsu et al., 2021; Liu et al., 2020c,b; Ravanelli et al., 2020; Ling et al., 2020; Ling and Liu, 2020). Self-supervised learning (SSL) is the main driver of this paradigm, an effective and scalable way to learn high-level representation of language that transfers to a vari- ety of tasks. SSL entails learning from the input or some perturbation of it without the need for la- belled data. This has unlocked the usage of large amounts of cheaply available unlabelled data. It lends naturally to neural network models that have been shown to possess impressive scaling charac- teristics such that it is often enough to increase the model and data sizes to improve …

Content truncated to 500 words