MASSIVE: A 1M-Example Multilingual Natural Language Understanding Dataset with 51 Typologically-Diverse Languages

Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics Volume 1: Long Papers, pages 4277–4302 July 9-14, 2023 ©2023 Association for Computational Linguistics MASSIVE: A 1M-Example Multilingual Natural Language Understanding Dataset with 51 Typologically-Diverse Languages Jack FitzGerald ∗1 Christopher Hench1 Charith Peris1 Scott Mackie1 Kay Rottmann1 Ana Sanchez1 Aaron Nash1 Liam Urbach1 Vishesh Kakarala1 Richa Singh1 Swetha Ranganath2 Laurie Crist3 Misha Britan1 Wouter Leeuwis1 Gokhan Tur1 Prem Natarajan4 1Amazon 2Microsoft 3Tripadvisor 4Capital One Abstract We present the MASSIVE dataset— Multilingual Amazon Slu resource package (SLURP) for Slot-filling, Intent classification, and Virtual assistant Evaluation. MASSIVE contains 1M realistic, parallel, labeled virtual assistant utterances spanning 51 languages, 18 domains, 60 intents, and 55 slots. MASSIVE was created by tasking professional translators to localize the English-only SLURP dataset into 50 typologically diverse languages from 29 genera. We also present modeling results on XLM-R and mT5, including exact match accuracy, intent classification accuracy, and slot-filling F1 score. We have released our dataset, modeling code, and models publicly. 1 Introduction and Description Natural Language Understanding (NLU) is a machine’s ability to understand the meaning and relevant entities from text. For instance, given the utterance what is the temperature in new york, an NLU model might classify the intent as weather_query and fill the slots as weather_descriptor: temperature and place_name: new york. Our partic- ular focus of NLU is one component of Spoken Language Understanding (SLU), in which raw audio is first converted to text before NLU is performed (Young, 2002; Wang et al., 2005; Tur and Mori, 2011). SLU is the foundation of voice-based virtual assistants like Alexa, Siri, and Google Assistant. Though virtual assistants have advanced incredibly in the past decade, they still only support a small fraction of the world’s 7,000+ languages (Simons, 2022). Challenges *Corresponding author, jgmf@amazon.com. All au- thors were associated with Amazon at the time of publication. for multilingualism span the software stack and a variety of operational considerations, but one difficulty in creating massively multilingual NLU models is the lack of labeled data for training and evaluation, particularly data that is realistic for the task and that is natural for each given language. High naturalness typically requires human-based vetting, which is often costly. We present MASSIVE (Multilingual Amazon SLU Resource Package (SLURP) for Slot fill- ing, Intent classification, and Virtual assistant Evaluation), a new 1M-example dataset composed of realistic, human-created virtual assistant utter- ance text spanning 51 languages, 60 intents, 55 slot types, and 18 domains. With the English seed data included, there are 587k train utterances, 104k dev utterances, 152k test utterances, and 153k ut- terances currently held out for the MMNLU-22 competition, which will be released after the com- petition. We have released our data, code, and models 1. MASSIVE was created by localizing the SLURP NLU dataset (created only in English) in a parallel manner. SLURP is described further in Section 2, linguistic analyses of the dataset in Section 3, and the localization process in Section 4.3. Results for Massively Multilingual NLU (MMNLU) …

Content truncated to 500 words