Sliced at SemEval-2022 Task 11: Bigger, Better? Massively Multilingual LMs for Multilingual Complex NER on an Academic GPU Budget

Proceedings of the 16th International Workshop on Semantic Evaluation (SemEval-2022), pages 1494 - 1500 July 14-15, 2022 ©2022 Association for Computational Linguistics Sliced at SemEval-2022 Task 11: Bigger, Better? Massively Multilingual LMs for Multilingual Complex NER on an Academic GPU Budget Barbara Plank Center for Information and Language Processing (CIS), LMU Munich, Germany Department of Computer Science, ITU Copenhagen, Denmark bplank@itu.dk Abstract Massively multilingual language models (MMLMs) have become a widely-used language representation method, and multiple large MMLMs were proposed in recent years. A trend is to train MMLMs on larger text corpora or with more layers. In this paper we set out to evaluate recent popular MMLMs on detecting semantically ambiguous and complex named entities with an academic GPU budget. Our submission of a single model for 11 languages on the SemEval Task 11 MultiCoNER shows that a fine-tuned XLM-Rlarge outperforms the more recent RemBERT, ranking 9th from 26 submissions in the multilingual track. Compared to Rem- BERT, the XLM-R model has the additional advantage to fit on a slice of a multi-instance GPU. As contrary to expectations and recent findings, we found RemBERT to not be the best MMLM, we further set out to investigate this discrepancy with additional experiments on multilingual Wikipedia NER data. While we expected RemBERT to have an edge on that dataset as it is closer to its pre-training data, surprisingly, our results show that this is not the case. 1 Introduction Pre-trained language models have revolutionized the field of Natural Language Processing (NLP) in recent years (Peters et al., 2018; Devlin et al., 2019; Zhuang et al., 2021). Especially for cross-lingual transfer learning or creating a single multilingual model, pre-trained massively multilingual language models (MMLMs) have become a de-facto stan- dard (Conneau et al., 2020; Chung et al., 2021). MMLMs such as BERT and XLM-R share the same underlying idea: multilingual representations are obtained by learning from large text collections in multiple languages and are trained using a lan- guage modeling objective. MMLMs, in contrast to alternative cross-lingual transfer strategies, thus do not rely on explicit alignment via parallel data and explicit transfer via translations with e.g. anno- tation projection. MMLMs, together with the pre- training and fine-tuning paradigm, have enabled impressive results (Conneau et al., 2020; Lauscher et al., 2020; Müller-Eberstein et al., 2021). This paper describes our submission to Task 11 on Multilingual Complex Named Entity Recogni- tion (MultiCoNER) (Malmasi et al., 2022b,a). We evaluate several recent MMLMs in a fine-tuning regime to answer the following main research ques- tion (RQ1): To what extent are more recent larger LMs outperforming earlier MMLMs for the task of multilingual complex NER? To do so, we test four MMLMs (mBERT, XLM-R base and large and the most recently proposed RemBERT). As other NER datasets exists, albeit with different labels, we fur- ther explore multi-task learning (RQ2): To what extent can we improve upon MultiCoNER by using cross-lingual cross-domain NER data as auxiliary data? Finally, as the MMLMs were pre-trained on different kinds of data, …

Content truncated to 500 words