On the Economics of Multilingual Few-shot Learning: Modeling the Cost-Performance Trade-offs of Machine Translated and Manual Data

Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 1369 - 1384 July 10-15, 2022 ©2022 Association for Computational Linguistics On the Economics of Multilingual Few-shot Learning: Modeling the Cost-Performance Trade-offs of Machine Translated and Manual Data Kabir Ahuja1 Monojit Choudhury2 Sandipan Dandapat2 1 Microsoft Research, India 2 Microsoft IDC {t-kabirahuja,monojitc,sadandap}@microsoft.com Abstract Borrowing ideas from Production functions in micro-economics, in this paper we introduce a framework to systematically evaluate the per- formance and cost trade-offs between machine- translated and manually-created labelled data for task-specific fine-tuning of massively mul- tilingual language models. We illustrate the effectiveness of our framework through a case- study on the TyDIQA-GoldP dataset. One of the interesting conclusions of the study is that if the cost of machine translation is greater than zero, the optimal performance at least cost is always achieved with at least some or only manually-created data. To our knowledge, this is the first attempt towards extending the con- cept of production functions to study data col- lection strategies for training multilingual mod- els, and can serve as a valuable tool for other similar cost vs data trade-offs in NLP. 1 Introduction Transformer based Massively Multilingual Lan- guage Models (MMLMs) such as mBERT (De- vlin et al., 2019) , XLM-RoBERTa (Conneau et al., 2020) and mT5 (Xue et al., 2021) are surprisingly effective at zero-shot cross-lingual transfer (Pires et al., 2019; Wu and Dredze, 2019). However, while zero-shot transfer is effective, often the per- formances across different languages is not con- sistent. Low-resource languages (Wu and Dredze, 2020) and the languages that are typologically dis- tant from the pivot language (Lauscher et al., 2020) are known to benefit the least from zero-shot trans- fer, which can often be mitigated by using target- language specific labelled data for the task during fine-tuning. One common approach for collecting such data in the target language is to translate the training data available for the pivot-language to the target by using an off-the-shelf Machine Translation (MT) system. This is usually referred to as the translate- train setup (Hu et al., 2020; Turc et al., 2021). Few-shot transfer is another alternative; as shown by Lauscher et al. (2020), a few labelled examples in the target language, that can be obtained cheaply, can lead to substantial improvements over the zero- shot performance. However, there has not been much work on com- paring the performance across these two strategies. In one such study, Hu et al. (2020) compare the performance of translate-train with few-shot trans- fer on TyDIQA-GoldP (Clark et al., 2020) dataset, but they only evaluate the few-shot case with 1000 examples, which does not provide any insight into how the performance varies with increasing dataset sizes for these two approaches. Additionally, there are trade-offs related to the data acquisition costs as well. The cost per training instance is expected to be much smaller for an MT-based approach than manual translation or labeling of examples. How- ever, depending on …

Content truncated to 500 words