Probing Task-Oriented Dialogue Representation from Language Models

Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing, pages 549–554, November 16–20, 2020. c⃝2020 Association for Computational Linguistics 549 Don’t Use English Dev: On the Zero-Shot Cross-Lingual Evaluation of Contextual Embeddings Phillip Keung Yichao Lu Julian Salazar Vikas Bhardwaj Amazon {keung,yichaolu,julsal,vikab}@amazon.com Abstract Multilingual contextual embeddings have demonstrated state-of-the-art performance in zero-shot cross-lingual transfer learning, where multilingual BERT is fine-tuned on one source language and evaluated on a different target language. However, published results for mBERT zero-shot accuracy vary as much as 17 points on the MLDoc classification task across four papers. We show that the standard practice of using English dev accuracy for model selection in the zero-shot setting makes it difficult to obtain reproducible results on the MLDoc and XNLI tasks. English dev accuracy is often uncorrelated (or even anti-correlated) with target language accuracy, and zero-shot performance varies greatly at different points in the same fine-tuning run and between different fine-tuning runs. These reproducibility issues are also present for other tasks with different pre-trained embeddings (e.g., MLQA with XLM-R). We recommend providing oracle scores alongside zero-shot results: still fine-tune using English data, but choose a checkpoint with the target dev set. Reporting this upper bound makes results more consistent by avoiding arbitrarily bad checkpoints. 1 Introduction Zero-shot and zero-resource cross-lingual NLP has seen significant progress in recent years. The dis- covery of cross-lingual structure in word embed- ding spaces culminated in the work of Lample et al. (2018b), which showed that unsupervised word translation via adversarial mappings is com- petitive with supervised techniques. Concurrent work in machine translation also showed that it is possible to achieve non-trivial BLEU scores with- out any bitext (Artetxe et al., 2018; Lample et al., 2018a). Self-supervised multilingual contextual embeddings like mBERT (Devlin et al., 2019) and XLM (Conneau and Lample, 2019) have shown remarkably strong performance on cross-lingual named entity recognition, text classification, depen- dency parsing, and other tasks (e.g., Pires et al., 2019; Keung et al., 2019; Wu and Dredze, 2019). Much of this recent work has demonstrated that mBERT performs very well on zero-shot tasks, su- perseding prior techniques as the baseline for zero- shot cross-lingual transfer learning. By zero-shot, we mean that no parallel text or labeled data from the target language was used during model train- ing, fine-tuning, or hyperparameter search. In this setting, models are trained on labeled (usually En- glish) text and tested on target (non-English) text. Standard practice prohibits the use of target lan- guage data for model selection; the final model is chosen using the English dev set only. However, we find that zero-shot mBERT results can vary greatly. We present 4 published baselines for zero-shot cross-lingual document classification on MLDoc (Schwenk and Li, 2018) in Table 1: MLDoc En Fr Ja Ru Zh Eisenschlos et al. (2019) 93.2 83.0 64.6 71.6 66.2 Dong and de Melo (2019) 94.2 80.0 73.2 70.7 75.4 Keung et al. (2019) 94.2 73.5 72.8 73.7 76.0 Wu and Dredze (2019) 94.2 72.6 56.5 73.7 76.9 …

Content truncated to 500 words