Data Augmentation for Voice-Assistant NLU using BERT-based Interchangeable Rephrase

Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics, pages 1852–1860 April 19 - 23, 2021. ©2021 Association for Computational Linguistics 1852 Data Augmentation for Voice-Assistant NLU using BERT-based Interchangeable Rephrase Akhila Yerukola∗, Mason Bretan∗, Hongxia Jin Samsung Research America {a.yerukola, mason.bretan, hongxia.jin}@samsung.com Abstract We introduce a data augmentation technique based on byte pair encoding and a BERT- like self-attention model to boost performance on spoken language understanding tasks. We compare and evaluate this method with a range of augmentation techniques encompass- ing generative models such as VAEs and performance-boosting techniques such as syn- onym replacement and back-translation. We show our method performs strongly on domain and intent classification tasks for a voice assis- tant and in a user-study focused on utterance naturalness and semantic similarity. 1 Introduction With conversational assistants becoming more and more pervasive in everyday life, task-oriented dia- logue systems are rapidly evolving. These systems typically consist of Spoken Language Understand- ing (SLU) models, tasked with determining the domain category and the intent of user utterances at each turn of the conversation. The ability to quickly train such models to meet changing and evolving user needs is necessary. However, developers often find themselves in sit- uations with access to very little labeled training data. This is especially true when new functions are deployed, and a large user base has not had a chance to utilize the function, thus, limiting the number of available utterances that can be labeled for training. Furthermore, the process of labeling large amounts of data can be time consuming and expensive. More recently, these challenges have been both enhanced and complicated by privacy concerns and legislation that may prevent the use of user utterances for training. Much of the recent research addressing data paucity has focused on pre-training using self- supervision and vast amounts of unlabeled data ∗equal contribution Utterance how do I make a margherita pizza Domain Recipe Intent Return Recipe Rephrase show me how to cook a margherita pizza Utterance can you check air quality in santa rosa Domain Weather Intent Air Quality Rephrase is it possible to check air quality in santa rosa Utterance delete all emails which have come from hotel.com Domain Email Intent Delete Emails Rephrase get rid of all emails which have come from hotel.com Table 1: An illustrative example of a generated rephrase using our Interchangeable Rephrase, while still maintaining the original domain and intent. (Devlin et al., 2018; Radford et al., 2018, 2019). Pre-trained models can later be fine-tuned with a much smaller amount of labeled data for specific tasks. In this work, instead of pre-training, we explore methods that enhance and expand the task- specific training set by using data augmentation. While models such as BERT prove to be both use- ful and relevant, we show that data augmentation during the fine-tuning stage can boost performance even on these large pre-trained models. We imple- ment and compare several pre-existing techniques for data augmentation on Natural Language Under- standing …

Content truncated to 500 words