Improving Zero-Shot Multilingual Text Generation via Iterative Distillation

Proceedings of the 29th International Conference on Computational Linguistics, pages 5876–5881 October 12–17, 2022. 5876 Improving Zero-Shot Multilingual Text Generation via Iterative Distillation †Ernie Chang, *Alex Marin, †Vera Demberg †Dept. of Language Science and Technology, Saarland University *Microsoft Corporation, Redmond, WA cychang@coli.uni-saarland.de Abstract The demand for multilingual dialogue systems often requires a costly labeling process, where human translators derive utterances in low re- source languages from resource rich language annotation. To this end, we explore leverag- ing the inductive biases for target languages learned by numerous pretrained teacher mod- els by transferring them to student models via sequence-level knowledge distillation. By as- suming no target language text, both the teacher and student models need to learn from the tar- get distribution in a few/zero-shot manner. On the MultiATIS++ benchmark, we explore the effectiveness of our proposed technique to de- rive the multilingual text for 6 languages, using only the monolingual English data and the pre- trained models. We show that training on the synthetic multilingual generation outputs yields close performance to training on human anno- tations in both slot F1 and intent accuracy; the synthetic text also scores high in naturalness and correctness based on human evaluation. 1 Introduction In multilingual dialogue systems, natural language generation is used to generate utterances in various languages, using as input semantic frames, which contain a representation of the user intent together with relevant information or entities related to said intent (Tur et al., 2010). Past works that general- ize dialogue systems to multilingual settings often made two unrealistic assumptions about the data availability of any new dialogue domain (Liu et al., 2019; Xu et al., 2020; Schuster et al., 2019; Chang et al., 2020, 2022): (1) First assumption is that a large set of monolingual data has already been an- notated. (2) Some in-domain text or annotated data in the target languages are already available for the purpose of transfer learning. Neither assumption holds in all cases (Upadhyay et al., 2018). To overcome these challenges, we utilize knowledge-grounded pre-training (KGPT) (Chen MT Model E-Step mBART M-Step teacher student Frames KGPT English Text x x x x x x x x x De Multilingual Data Figure 1: Annotation scenario: Each × represents a labeled data instance. The goal is to generalize from few-shot human- labeled instances in one language (left) to large synthetic multilingual data (right). et al., 2020) – a pretrained data-to-text generation model that was shown to be effective in overcom- ing data scarcity, as the model is capable of rep- resenting the inductive biases required to encode structured data such as the slot-value pairs (frames). In few-shot settings, we can exploit KGPT’s pre- trained knowledge to obtain in-domain text la- bels for a large set of unlabeled frame sequences. These text labels are then converted to multiple lan- guages with the use of bilingual translation mod- els as teacher models, as inspired by past works on sequence-level knowledge distillation (Wang et al., 2020; Kim and Rush, 2016; Gordon and Duh, 2019). …

Content truncated to 500 words