From Masked Language Modeling to Translation: Non-English Auxiliary Tasks Improve Zero-shot Spoken Language Understanding

Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 2479–2497 June 6–11, 2021. ©2021 Association for Computational Linguistics 2479 From Masked Language Modeling to Translation: Non-English Auxiliary Tasks Improve Zero-shot Spoken Language Understanding Rob van der Goot1, Ibrahim Sharaf2, Aizhan Imankulova3, Ahmet ¨Ust¨un4, Marija Stepanovi´c1, Alan Ramponi5,6, Siti Oryza Khairunnisa3, Mamoru Komachi3, Barbara Plank1 1IT University of Copenhagen, 2Factmata, 3Tokyo Metropolitan University, 4University of Groningen, 5University of Trento, 6Fondazione The Microsoft Research – University of Trento Centre for Computational and Systems Biology (COSBI) {robv,bapl}@itu.dk Abstract The lack of publicly available evaluation data for low-resource languages limits progress in Spoken Language Understanding (SLU). As key tasks like intent classification and slot fill- ing require abundant training data, it is desir- able to reuse existing data in high-resource languages to develop models for low-resource scenarios. We introduce XSID, a new bench- mark for cross-lingual (X) Slot and Intent De- tection in 13 languages from 6 language fam- ilies, including a very low-resource dialect. To tackle the challenge, we propose a joint learning approach, with English SLU training data and non-English auxiliary tasks from raw text, syntax and translation for transfer. We study two setups which differ by type and lan- guage coverage of the pre-trained embeddings. Our results show that jointly learning the main tasks with masked language modeling is effec- tive for slots, while machine translation trans- fer works best for intent classification.1 1 Introduction Digital conversational assistants have become an integral part of everyday life and they are avail- able, e.g., as standalone smart home devices or in smartphones. Key steps in such task-oriented con- versational systems are recognizing the intent of a user’s utterance, and detecting the main arguments, also called slots. For example, for an utterance like “Add reminder to swim at 11am tomorrow”, these key Natural Language Understanding (NLU), or Spoken Language Understanding (SLU) tasks are illustrated in Figure 1. As slots depend on the intent type, leading models typically adopt joint solutions (Chen et al., 2019; Qin et al., 2020). Despite advances in neural modeling for slot and intent detection (§ 6), datasets for SLU re- main limited, hampering progress toward provid- ing SLU for many language varieties. Most avail- 1The source code, dataset and predictions are available at: https://bitbucket.org/robvanderg/xsid Add reminder to swim at 11am tomorrow intent: add reminder Figure 1: English example from XSID annotated with intents (add reminder) and slots ( todo , datetime ). The full set of languages is shown in Table 2. able datasets either support only a specific domain (like air traffic systems) (Xu et al., 2020), or are broader but limited to English and a few other lan- guages (Schuster et al., 2019; Coucke et al., 2018). We release XSID, a new benchmark intended for SLU evaluation in low-resource scenarios. XSID contains evaluation data for 13 languages from six language families, including a very low-resource dialect. It homogenizes annotation styles of two recent datasets (Schuster et al., …

Content truncated to 500 words