SLURP: A Spoken Language Understanding Resource Package

Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing, pages 7252–7262, November 16–20, 2020. c⃝2020 Association for Computational Linguistics 7252 SLURP: A Spoken Language Understanding Resource Package Emanuele Bastianelli†∗, Andrea Vanzo†∗, Pawel Swietojanski‡∗ and Verena Rieser† †The Interaction Lab, MACS, Heriot-Watt University, Edinburgh, UK ‡Faculty of Engineering, University of New South Wales, Sydney, Australia {e.bastianelli, a.vanzo, v.t.rieser}@hw.ac.uk p.swietojanski@unsw.edu.au Abstract Spoken Language Understanding infers se- mantic meaning directly from audio data, and thus promises to reduce error propagation and misunderstandings in end-user applica- tions. However, publicly available SLU re- sources are limited. In this paper, we release SLURP, a new SLU package containing the following: (1) A new challenging dataset in English spanning 18 domains, which is sub- stantially bigger and linguistically more di- verse than existing datasets; (2) Competitive baselines based on state-of-the-art NLU and ASR systems; (3) A new transparent metric for entity labelling which enables a detailed er- ror analysis for identifying potential areas of improvement. SLURP is available at https: //github.com/pswietojanski/slurp 1 Introduction Traditionally, Spoken Language Understanding (SLU) uses a pipeline transcribing audio into text using Automatic Speech Recognition (ASR), which is then mapped into a semantic structure via Natural Language Understanding (NLU). However, this modular approach is prone to error propaga- tion from noisy ASR transcriptions, and ASR in turn is not able to disambiguate based on seman- tic information. End-to-end (E2E) approaches on the other hand, can benefit from joint modelling. One of the main bottlenecks for building E2E-SLU systems, however, is the lack of large and diverse datasets of audio inputs paired with correspond- ing semantic structures. Publicly available datasets to date are limited in terms of lexical and seman- tic richness (Lugosch et al., 2019b), number of vocalizations (Coucke et al., 2018), domain cover- age (Hemphill et al., 1990; Dahl et al., 1994) and semantic contexts (Godfrey et al., 1992; Jurafsky and Shriberg, 1997). In this paper, we present the ∗Authors contributed equally. User: “Make a calendar entry for brunch on Satur- day morning with Aaronson.” Scenario: Calendar Action: Create entry Entity tags and lexical fillers: [event name: brunch], [date: Saturday], [timeofday: morning], [person: Aaronson] Figure 1: Example annotation from SLURP dataset. Spoken Language Understanding Resource Pack- age (SLURP), a publicly available multi-domain dataset for E2E-SLU, which is substantially big- ger and more diverse than existing SLU datasets. SLURP is a collection of ~72k audio recordings of single turn user interactions with a home as- sistant, annotated with three levels of semantics: Scenario, Action and Entities, as in Fig. 1, includ- ing over 18 different scenarios, with 46 defined actions and 55 different entity types as listed on https://github.com/pswietojanski/slurp.1 In order to further support SLU development, we propose SLU-F1, a new metric for entity prediction, which is specifically designed to assess error prop- agation in structured E2E-SLU tasks. This metric has 3 main advantages over the commonly used ac- curacy/F1 metric, aimed at supporting SLU devel- opers: First, it computes a distribution rather than a single score. This distribution is …

Content truncated to 500 words