To What Degree Can Language Borders Be Blurred In BERT-based Multilingual Spoken Language Understanding?

Proceedings of the 28th International Conference on Computational Linguistics, pages 2699–2709 Barcelona, Spain (Online), December 8-13, 2020 2699 To What Degree Can Language Borders Be Blurred In BERT-based Multilingual Spoken Language Understanding? Quynh Do, Judith Gaspers, Tobias R¨oding, Melanie Bradford Amazon Alexa {doquynh,gaspers,rodingtr,neunerm}@amazon.com Abstract This paper addresses the question as to what degree a BERT-based multilingual Spoken Language Understanding (SLU) model can transfer knowledge across languages. Through experiments we will show that, although it works substantially well even on distant language groups, there is still a gap to the ideal multilingual performance. In addition, we propose a novel BERT-based adversarial model architecture to learn language-shared and language-specific representations for multilingual SLU. Our experimental results prove that the proposed model is capable of narrowing the gap to the ideal multilingual performance. 1 Introduction Recently, modern voice-controlled devices with virtual assistants like Alexa, Siri, and Google Assistant, have been working their way into every activity of our daily life from playing a song to driving a car. Spoken Language Understanding (SLU) models interpreting the semantic meaning conveyed by an user’s spoken utterance, are the AI brains of these devices. The task of SLU is usually divided into two sub-tasks, namely intent classification (IC), which identifies the intent of an user’s utterance; and slot filling (SF), which extracts semantic constituents from the utterance. Consider an annotated utterance from the ATIS dataset (Price, 1990): city [O where] [O is] [B−airport code MCO] The SF sub-task should classify “where” and “is” as O (i.e. Other); and “MCO” as B − airport code, while the IC sub-task should identify city as the intent. Current state-of-the-art SLU models are mostly DNN-based joint models of the two sub-tasks (Liu and Lane, 2016; Do and Gaspers, 2019b; Chen et al., 2019a). These models usually contain two individual decoders to detect slot and intent labels on top of a shared encoder. As in many other NLP fields, the BERT encoder-decoder architecture has shown its strong ability in capturing contextual information from data sources to improve SLU performance (Chen et al., 2019a). For common SLU use cases, the BERT encoder is pre-trained on a huge amount of unlabeled texts and then fine-tuned on SLU training data. Since the voice-controlled device market has been expanding at an incredible rate all over the world, there is a rising need of fast language expansion for SLU models. Multilingual technology, which allows the development of a single model for multiple languages, and transferring knowledge from a data resource to languages other than its own, is currently one of the best solutions to meet the increased need. Traditionally, an SLU model is trained on supervised monolingual data leading to the fact that a virtual assistant often can use only one language within a working session. In contrast, a multilingual SLU model trained on supervised multilingual data, can provide the virtual assistant the ability to talk in multiple languages within a working session. Moreover, from a development perspective, multilingual modeling helps not only to reduce the number …

Content truncated to 500 words