SLUE Phase-2: A Benchmark Suite of Diverse Spoken Language Understanding Tasks

Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics Volume 1: Long Papers, pages 8906–8937 July 9-14, 2023 ©2023 Association for Computational Linguistics SLUE Phase-2: A Benchmark Suite of Diverse Spoken Language Understanding Tasks Suwon Shon1, Siddhant Arora2∗, Chyi-Jiunn Lin3∗, Ankita Pasad4∗ , Felix Wu1, Roshan Sharma2,Wei-Lun Wu3, Hung-Yi Lee3, Karen Livescu4, Shinji Watanabe2 1ASAPP 2Carnegie Mellon University 3National Taiwan University 4Toyota Technological Institute at Chicago Abstract Spoken language understanding (SLU) tasks have been studied for many decades in the speech research community, but have not re- ceived as much attention as lower-level tasks like speech and speaker recognition. In this work, we introduce several new annotated SLU benchmark tasks based on freely available speech data, which complement existing bench- marks and address gaps in the SLU evaluation landscape. We contribute four tasks: question answering and summarization involve infer- ence over longer speech sequences; named en- tity localization addresses the speech-specific task of locating the targeted content in the sig- nal; dialog act classification identifies the func- tion of a given speech utterance. In order to facilitate the development of SLU models that leverage the success of pre-trained speech rep- resentations, we will release a new benchmark suite, including for each task (i) curated an- notations for a relatively small fine-tuning set, (ii) reproducible pipeline (speech recognizer + text model) and end-to-end baseline models and evaluation metrics, (iii) baseline model per- formance in various types of systems for easy comparisons. We present the details of data collection and annotation and the performance of the baseline models. We also analyze the sensitivity of pipeline models’ performance to the speech recognition accuracy, using more than 20 publicly available speech recognition models. 1 Introduction Spoken language understanding (SLU) tasks in- volve inferring the linguistic structure or semantic meaning of a speech signal beyond its text tran- script. We use this term broadly to include any natural language processing (NLP) task applied to speech, and tasks that involve linguistic under- standing but also localization in the signal of rele- vant segments or producing speech as output. SLU ∗Core contributors in alphabetical order has been an active area throughout the history of speech research (Hemphill et al., 1990; Calhoun et al., 2010; Busso et al., 2008; Zadeh et al., 2018; Chen et al., 2020a; Cohn et al., 2019; Yadav et al., 2020; Martinez-Lucas et al., 2020). However, com- pared to "lower-level" tasks like automatic speech recognition (ASR) and speaker identification, SLU has received much less attention and resources, and specifically there are much fewer benchmarks with freely available data. SLU tasks can in principle be addressed via a pipeline approach — using ASR to map speech to text and an NLP (text) model to map text to the desired output. The alternative is an end-to- end (E2E) model, which maps directly from the input speech to the target output. While pipeline approaches can take advantage of existing strong ASR and NLP models, E2E models can be more efficient at inference time, can …

Content truncated to 500 words