Cross-lingual Intermediate Fine-tuning improves Dialogue State Tracking

Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 1137–1150 November 7–11, 2021. c⃝2021 Association for Computational Linguistics 1137 Cross-lingual Intermediate Fine-tuning improves Dialogue State Tracking Nikita Moghe and Mark Steedman and Alexandra Birch School of Informatics, University of Edinburgh {nikita.moghe, a.birch}@ed.ac.uk , steedman@inf.ed.ac.uk Abstract Recent progress in task-oriented neural dia- logue systems is largely focused on a hand- ful of languages, as annotation of training data is tedious and expensive. Machine trans- lation has been used to make systems mul- tilingual, but this can introduce a pipeline of errors. Another promising solution is us- ing cross-lingual transfer learning through pre- trained multilingual models. Existing methods train multilingual models with additional code- mixed task data or refine the cross-lingual rep- resentations through parallel ontologies. In this work, we enhance the transfer learning process by intermediate fine-tuning of pre- trained multilingual models, where the mul- tilingual models are fine-tuned with different but related data and/or tasks. Specifically, we use parallel and conversational movie subti- tles datasets to design cross-lingual interme- diate tasks suitable for downstream dialogue tasks. We use only 200K lines of parallel data for intermediate fine-tuning which is al- ready available for 1782 language pairs. We test our approach on the cross-lingual dia- logue state tracking task for the parallel Mul- tiWoZ (English→Chinese, Chinese→English) and Multilingual WoZ (English→German, English→Italian) datasets. We achieve impres- sive improvements (> 20% on joint goal ac- curacy) on the parallel MultiWoZ dataset and the Multilingual WoZ dataset over the vanilla baseline with only 10% of the target language task data and zero-shot setup respectively. 1 Introduction In recent years, task-oriented dialogue systems have achieved remarkable success by leveraging huge amounts of labelled data. This technology is thus limited to a handful of languages as collecting and annotating training dialogue data for different languages is expensive and requires supervision from native speakers (Chen et al., 2018). To avoid having to create large annotated datasets for every new language, recent works fo- cus on transfer learning methods which use neural machine translation systems (Schuster et al., 2019), code-mixed data augmentation (Liu et al., 2020; Qin et al., 2020) or large multilingual models (Lin and Chen, 2021). Neural machine translation mod- els incur additional overhead of training on millions of parallel sentences that may not be available for all language pairs. Code-mixed data augmenta- tion methods involve replacing individual words from the source language with the target language by using parallel word pairs found in a dictionary. However, a simple synonym replacement may not be sufficient as the tasks become complicated. In this paper, we focus on transfer learning via large multilingual models, which will allow us to extend models to languages with limited labelled training data. In techniques that use multilingual models, a task-specific architecture uses this pretrained model as one of its components and then is trained with task data from a high resource language (See Fig. 1). It is then evaluated directly or with some la- belled examples in a …

Content truncated to 500 words