Adversarial Adaptation of Synthetic or Stale Data

Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics, pages 1297–1307 Vancouver, Canada, July 30 - August 4, 2017. c⃝2017 Association for Computational Linguistics https://doi.org/10.18653/v1/P17-1119 Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics, pages 1297–1307 Vancouver, Canada, July 30 - August 4, 2017. c⃝2017 Association for Computational Linguistics https://doi.org/10.18653/v1/P17-1119 Adversarial Adaptation of Synthetic or Stale Data Young-Bum Kim† Karl Stratos‡ Dongchan Kim† †Microsoft AI and Research ‡Bloomberg L. P. {ybkim, dongchan.kim}@microsoft.com me@karlstratos.com Abstract Two types of data shift common in prac- tice are 1. transferring from synthetic data to live user data (a deployment shift), and 2. transferring from stale data to current data (a temporal shift). Both cause a dis- tribution mismatch between training and evaluation, leading to a model that over- fits the flawed training data and performs poorly on the test data. We propose a solu- tion to this mismatch problem by framing it as domain adaptation, treating the flawed training dataset as a source domain and the evaluation dataset as a target domain. To this end, we use and build on several recent advances in neural domain adap- tation such as adversarial training (Ganin et al., 2016) and domain separation net- work (Bousmalis et al., 2016), proposing a new effective adversarial training scheme. In both supervised and unsupervised adap- tation scenarios, our approach yields clear improvement over strong baselines. 1 Introduction Spoken language understanding (SLU) systems analyze various aspects of a user query by clas- sifying its domain, intent, and semantic slots. For instance, the query how is traffic to target in bellevue has domain PLACES, intent CHECK ROUTE TRAFFIC, and slots PLACE NAME: target and ABSOLUTE LOCATION: bellevue. We are interested in addressing two types of data shift common in SLU applications. The first data shift problem happens when we trans- fer from synthetic data to live user data (a deploy- ment shift). This is also known as the “cold-start” problem; a model cannot be trained on the real usage data prior to deployment simply because it does not exist. A common practice is to gener- ate a large quantity of synthetic training data that mimics the expected user behavior. Such synthetic data is crafted using domain-specific knowledge and can be time-consuming. It is also flawed in that it typically does not match the live user data generated by actual users; the real queries submit- ted to these systems are different from what the model designers expect to see. The second data shift problem happens when we transfer from stale data to current data (a tem- poral shift). In our use case, we have one set of training data from 2013 and wish to handle data from 2014–2016. This is problematic since the content of the user queries changes over time (e.g., new restaurant or movie names may be added). Consequently, the model performance degrades over time. Both shifts cause a distribution mismatch be- tween training and evaluation, leading to a model that overfits the flawed training data …

Content truncated to 500 words