Imitation Attacks and Defenses for Black-box Machine Translation Systems

Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing, pages 5531–5546, November 16–20, 2020. c⃝2020 Association for Computational Linguistics 5531 Imitation Attacks and Defenses for Black-box Machine Translation Systems Eric Wallace Mitchell Stern Dawn Song UC Berkeley {ericwallace, mitchell, dawnsong}@berkeley.edu Abstract Adversaries may look to steal or attack black- box NLP systems, either for financial gain or to exploit model errors. One setting of par- ticular interest is machine translation (MT), where models have high commercial value and errors can be costly. We investigate pos- sible exploitations of black-box MT systems and explore a preliminary defense against such threats. We first show that MT systems can be stolen by querying them with monolingual sen- tences and training models to imitate their out- puts. Using simulated experiments, we demon- strate that MT model stealing is possible even when imitation models have different input data or architectures than their target models. Applying these ideas, we train imitation mod- els that reach within 0.6 BLEU of three produc- tion MT systems on both high-resource and low-resource language pairs. We then lever- age the similarity of our imitation models to transfer adversarial examples to the produc- tion systems. We use gradient-based attacks that expose inputs which lead to semantically- incorrect translations, dropped content, and vulgar model outputs. To mitigate these vul- nerabilities, we propose a defense that mod- ifies translation outputs in order to misdirect the optimization of imitation models. This defense degrades the adversary’s BLEU score and attack success rate at some cost in the de- fender’s BLEU and inference speed. 1 Introduction NLP models deployed through APIs (e.g., Google Translate) can be lucrative assets for an organiza- tion. These models are typically the result of a con- siderable investment—up to millions of dollars— into private data annotation and algorithmic im- provements. Consequently, such models are kept hidden behind black-box APIs to protect system integrity and intellectual property. We consider an adversary looking to steal or attack a black-box NLP system. Stealing a produc- tion model allows an adversary to avoid long-term API costs or launch a competitor service. More- over, attacking a system using adversarial exam- ples (Szegedy et al., 2014) allows an adversary to cause targeted errors for a model, e.g., bypassing fake news filters or causing systems to output ma- licious content that may offend users and reflect poorly on system providers. In this work, we in- vestigate these two exploits for black-box machine translation (MT) systems: we first steal (we use “steal” following Tram`er et al. 2016) production MT systems by training imitation models, and we then use these imitation models to generate adver- sarial examples for production MT systems. We create imitation models by borrowing ideas from knowledge distillation (Hinton et al., 2014): we query production MT systems with monolingual sentences and train imitation (i.e., student) mod- els to mimic the system outputs (top of Figure 1). We first experiment with simulated studies which demonstrate that MT models are easy to imitate (Section 3). For …

Content truncated to 500 words