SLICER: Sliced Fine-Tuning for Low-Resource Cross-Lingual Transfer for Named Entity Recognition

Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 10775 - 10785 December 7-11, 2022 ©2022 Association for Computational Linguistics SLICER: Sliced Fine-Tuning for Low-Resource Cross-Lingual Transfer for Named Entity Recognition Fabian David Schmidt1, Ivan Vuli´c2, Goran Glavaš1 1 Center For Artificial Intelligence and Data Science, University of Würzburg, Germany 2 Language Technology Lab, University of Cambridge, UK {fabian.schmidt, goran.glavas}@uni-wuerzburg.de iv250@cam.ac.uk Abstract Large multilingual language models generally demonstrate impressive results in zero-shot cross-lingual transfer, yet often fail to success- fully transfer to low-resource languages, even for token-level prediction tasks like named en- tity recognition (NER). In this work, we intro- duce a simple yet highly effective approach for improving zero-shot transfer for NER to low- resource languages. We observe that NER fine- tuning in the source language decontextualizes token representations, i.e., tokens increasingly attend to themselves. This increased reliance on token information itself, we hypothesize, triggers a type of overfitting to properties that NE tokens within the source languages share, but are generally not present in NE mentions of target languages. As a remedy, we propose a simple yet very effective sliced fine-tuning for NER (SLICER) that forces stronger token con- textualization in the Transformer: we divide the transformed token representations and classifier into disjoint slices that are then independently classified during training. We evaluate SLICER on two standard benchmarks for NER that in- volve low-resource languages, WikiANN and MasakhaNER, and show that it (i) indeed re- duces decontextualization (i.e., extent to which NE tokens attend to themselves), consequently (ii) yielding consistent transfer gains, especially prominent for low-resource target languages distant from the source language. 1 Introduction In recent years, massively multilingual transform- ers (MMTs) have become the backbone of multi- lingual NLP. MMTs like mBERT (Devlin et al., 2019), XLM-R (Conneau et al., 2020), and mT5 (Xue et al., 2021), pretrained on corpora spanning 100+ languages, have become the main vehicle of cross-lingual transfer in NLP: fine-tuned using task annotations in the source language, an MMT can, conceptually, directly make predictions for all tar- get languages seen in pretraining, for which little or no annotated task data exists (Pires et al., 2019; Wu and Dredze, 2019; Dufter and Schütze, 2020). Successful zero-shot transfer, however, has been shown to critically hinge on linguistic proximity between source and target languages as well the quality of target language representations, deter- mined by the size of target language corpora used in the MMT pretraining (Lauscher et al., 2020; Zhao et al., 2021). Unfortunately, the transfer fails where it is needed the most – for low-resource languages linguistically distant from high-resource languages with annotated task data (Ebrahimi et al., 2021; Adelani et al., 2021; Ruder et al., 2021b). Zero- shot transfer of named entity recognition (NER) models to low-resource languages suffers from a particularly profound performance drop (Adelani et al., 2021; Lauscher et al., 2020). In this work, we identify the cause and propose an effective remedy. Contributions. (1) We analyze the representation space of an MMT, before and after …

Content truncated to 500 words