InFoBERT: Zero-Shot Approach to Natural Language Understanding Using Contextualized Word Embedding

Proceedings of Recent Advances in Natural Language Processing, pages 208–215 Sep 1–3, 2021. https://doi.org/10.26615/978-954-452-072-4_025 208 InFoBERT: Zero-Shot Approach to Natural Language Understanding Using Contextualized Word Embedding Pavel Burnyshev Anfrey Bout Valentin Malykh Irina Piontkovskaya Huawei Noah’s Ark lab, Moscow Russia lastname.firstname@huawei.com Abstract Natural language understanding is an impor- tant task in modern dialogue systems. It be- comes more important with the rapid exten- sion of the dialogue systems’ functionality. In this work, we present an approach to zero-shot transfer learning for the tasks of intent clas- sification and slot-filling based on pre-trained language models. We use deep contextualized models feeding them with utterances and natu- ral language descriptions of user intents to get text embeddings. These embeddings then used by a small neural network to produce predic- tions for intent and slot probabilities. This ar- chitecture achieves new state-of-the-art results in two zero-shot scenarios. One is a single lan- guage new skill adaptation and another one is a cross-lingual adaptation. 1 Introduction Dialogue systems become more and more popular in everyday life. A dialogue system has two main language understanding tasks which are of key im- portance for any skill: user intent recognition and information extraction from user input. The former task traditionally is interpreted as a classification problem, since the possible intents list assumed to be given. The latter task is formulated as slot fill- ing, i.e. extraction of a text span considered to be a slot value. With dialogue systems becoming more popular their functionality is also growing in terms of both skills available to use and languages their functionality is accessible in. The rapid growth en- tails lack of usage data for the new skills and new languages and causes the lists of intents and slots to be dynamic thus make classical approaches inappli- cable. We consider these newly emerged scenarios and demonstrate that our approach is well suited for both. In the first scenario, there are several known domains and a few unknown domains. The second scenario contains skills with training data for one language (English) and only test data for two other languages (Spanish & Thai). In this paper, we present a new architecture which is based on pre-trained language models, like BERT (Devlin et al., 2019), and uses a small “head” (i.e. a neural network using the embeddings from the language models as input) evaluated in two scenarios: zero-shot single language natural language understanding on Schema-Guided Dia- log dataset (Rastogi et al., 2019) and zero-shot cross-lingual natural language understanding on Multi-lingual Task-oriented Dialog dataset (Schus- ter et al., 2019). For the Schema-Guided Dialog dataset, there is a baseline described alongside the dataset which is evaluated in the zero-shot trans- fer learning setup. It is using deep contextualized embedding model (BERT) to embed the utterances and natural language descriptions for the slots and intents. Another approach was presented in (Ruan et al., 2020), where the authors present a system based on the baseline with an addition of hand- crafted context features. Our model is also …

Content truncated to 500 words