A Multitask Active Learning Framework for Natural Language Understanding

Proceedings of the 28th International Conference on Computational Linguistics, pages 4900–4914 Barcelona, Spain (Online), December 8-13, 2020 4900 A Multitask Active Learning Framework for Natural Language Understanding Hua Zhu∗, Wu Ye∗, Sihan Luo∗, Xidong Zhang∗ vivo Mobile Communication Co., Ltd {zhuhua.bjai, yewu.aiyjy, sihan.luo, xidong.zhang}@vivo.com Abstract Natural language understanding (NLU) aims at identifying user intent and extracting seman- tic slots. This requires sufficient annotating data to get considerable performance in real-world situations. Active learning (AL) has been well-studied to decrease the needed amount of the annotating data and successfully applied to NLU. However, no research has been done on inves- tigating how the relation information between intents and slots can improve the efficiency of AL algorithms. In this paper, we propose a multitask AL framework for NLU. Our framework enables pool- based AL algorithms to make use of the relation information between sub-tasks provided by a joint model, and we propose an efficient computation for the entropy of a joint model. Experi- mental results show our framework can achieve competitive performance with less training data than baseline methods on all datasets. We also demonstrate that when using the entropy as the query strategy, the model with complete relation information can perform better than those with partial information. Additionally, we demonstrate that the efficiency of these active learning algorithms in our framework is still effective when incorporate with the Bidirectional Encoder Representations from Transformers (BERT). 1 Introduction Voice assistant becomes increasingly intelligent on the mobile phone. Given an utterance from the user, the task of natural language understanding (NLU) aims at identifying the user’s intent and extracting semantic information. This can help a voice assistant to convert the utterance to an executable instruction. The task can be separated as two sub-tasks called intent detection (ID) and slot filling (SF) (Tur and De Mori, 2011). Table 1 shows an example of ID and SF using query “Flights from Ontario to Orlando”. Query Flights from Ontario to Orlando Slots O O B-fromloc O B-toloc Intent atis flight Table 1: example from user query to semantic frame Traditional pipeline models solved the task individually (Haffner et al., 2003; Raymond and Riccardi, 2007; Yao et al., 2014). Considering possible error propagation effect caused by pipeline method, the joint model has been proposed to exploit the relation information between ID and SF (Liu and Lane, 2016; Goo et al., 2018; Chen et al., 2019a; Qin et al., 2019). Sufficient data becomes another bottleneck in real-world situations. Both ID and SF need a consid- erable amount of training data to get a satisfying performance. Acquiring annotated data is trivial and expensive. The Active Learning (AL) method offers a promising solution to deal with this bottleneck (Settles and Craven, 2008; Settles, 2009). It allows the model to query certain unlabeled samples to anno- tate for training. For multitask framework, researchers proposed methods to select samples considering This work is licensed under a Creative Commons Attribution 4.0 International License. License details: http:// creativecommons.org/licenses/by/4.0/ ∗Equal contributions. 4901 task priority or dependency …

Content truncated to 500 words