Federated Learning for Spoken Language Understanding

Proceedings of the 28th International Conference on Computational Linguistics, pages 3467–3478 Barcelona, Spain (Online), December 8-13, 2020 3467 Federated Learning for Spoken Language Understanding Zhiqi Huang1, Fenglin Liu1, Yuexian Zou1,2∗ 1ADSPLAB, School of ECE, Peking University, Shenzhen, China 2Peng Cheng Laboratory, Shenzhen, China {zhiqihuang, fenglinliu98, zouyx}@pku.edu.cn Abstract Recently, spoken language understanding (SLU) has attracted extensive research interests, and various SLU datasets have been proposed to promote the development. However, most of the existing methods focus on a single individual dataset, the efforts to improve the robustness of models and obtain better performance by combining the merits of various datasets are not well studied. In this paper, we argue that if these SLU datasets are considered together, different knowledge from different datasets could be learned jointly, and there are high chances to pro- mote the performance of each dataset. At the same time, we further attempt to prevent data leakage when unifying multiple datasets which, arguably, is more useful in an industry setting. To this end, we propose a federated learning framework, which could unify various types of datasets as well as tasks to learn and fuse various types of knowledge, i.e., text representations, from different datasets and tasks, without the sharing of downstream task data. The fused text representations merge useful features from different SLU datasets and tasks and are thus much more powerful than the original text representations alone in individual tasks. At last, in order to provide multi-granularity text representations for our framework, we propose a novel Multi-view Encoder (MV-Encoder) as the backbone of our federated learning framework. Experiments on two SLU benchmark datasets, including two tasks (intention detection and slot filling) and fed- erated learning settings (horizontal federated learning, vertical federated learning and federated transfer learning), demonstrate the effectiveness and universality of our approach. Specifically, we are able to get 1.53% improvement on the intent detection metric accuracy. And we could also boost the performance of a strong baseline by up to 5.29% on the slot filling metric F1. Fur- thermore, by leveraging BERT as an additional encoder, we establish new state-of-the-art results on SNIPS and ATIS datasets, where we get 99.33% and 98.28% in terms of accuracy on intent detection task as well as 97.20% and 96.41% in terms of F1 score on slot filling task, respectively. 1 Introduction In recent years, Spoken Language Understanding (SLU) technology, which typically involves intent detection and slot filling, plays a crucial part in goal-oriented dialogue systems. The intent detection is treated as an utterance classification problem, which can be modeled using various classifiers based on deep neural networks (Liu and Lane, 2016; Zhang et al., 2016a; Zhang et al., 2017; Xia et al., 2018). The slot filling task can be formulated as a sequence labeling problem, and one of the popular approaches with good performance is using CRF and CNN (Xu and Sarikaya, 2013). The slots represent the word-level information while the intent stands for the sentence-level information (E et al., 2019). These two tasks are normally considered as …

Content truncated to 500 words