Performance Prediction via Bayesian Matrix Factorisation for Multilingual Natural Language Processing Tasks

Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics, pages 1790–1801 May 2-6, 2023 ©2023 Association for Computational Linguistics Performance Prediction via Bayesian Matrix Factorisation for Multilingual Natural Language Processing Tasks Viktoria Schram Daniel Beck Trevor Cohn School of Computing and Information Systems The University of Melbourne vschram@student.unimelb.edu.au {d.beck,t.cohn}@unimelb.edu.au Abstract Performance prediction for Natural Language Processing (NLP) seeks to reduce the exper- imental burden resulting from the myriad of different evaluation scenarios, e.g., the combi- nation of languages used in multilingual trans- fer. In this work, we explore the framework of Bayesian matrix factorisation for performance prediction, as many experimental settings in NLP can be naturally represented in matrix format. Our approach outperforms the state- of-the-art in several NLP benchmarks, includ- ing machine translation and cross-lingual entity linking. Furthermore, it also avoids hyperpa- rameter tuning and is able to provide uncer- tainty estimates over predictions. 1 Introduction Natural language processing (NLP) is an empiri- cal discipline, with progress driven by a myriad of tasks, domains, computational models and datasets (Xia et al., 2020). Models are often developed to be applicable to a large number of languages. Given that there are more than 7000 languages spoken in the world (Haddow et al., 2022), evaluating the model performance of each possible NLP scenario leads to a combinatorial explosion and would re- quire excessive time for training and testing, as well as the significant costs of evaluation resource creation. This is very time consuming and compu- tationally expensive, especially for large models and massively multilingual applications (Xia et al., 2020; Sharir et al., 2020). To illustrate, for neural machine translation (MT) models, retraining and testing for each new lan- guage pair is required (Fan et al., 2021), which quickly increases the number of experiments (as shown in Figure 1). In those scenarios, whether the model is appropriate for the considered test languages is not clear upfront. Similarly, for cross- lingual transfer (TSF) tasks like parsing (Das and Sarkar, 2020) or part-of-speech (POS) tagging jpn kor marlit mkd nld nob ara bos bul ces cmn dan deu ell eng epo est eus Target Source Figure 1: Matrix representation showing a subset of the performance scores for Wiki MT (Schwenk et al., 2021). Black: Lowest performance scores. Light orange: High- est performance scores. Grey: No performance score available. The goal is to predict the performance of language pairs depicted by grey cells. (de Vries et al., 2022), the performance on all lan- guages of interest is only known after training and testing has been performed. This is particularly in- teresting for the development of NLP technologies for low-resource languages (de Vries et al., 2022). In this paper we propose a method for per- formance prediction which can solve the above problems through automatically estimating perfor- mance scores of machine learning models, thus avoiding the need for excessive training and test cy- cles, and data creation costs. The goal is to predict the performance of a model solely based on …

Content truncated to 500 words