Combating the Curse of Multilinguality in Cross-Lingual WSD by Aligning Sparse Contextualized Word Representations

Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 2459 - 2471 July 10-15, 2022 ©2022 Association for Computational Linguistics Combating the Curse of Multilinguality in Cross-Lingual WSD by Aligning Sparse Contextualized Word Representations Gábor Berend Institute of Informatics, University of Szeged berendg@inf.u-szeged.hu Abstract In this paper, we advocate for using large pre- trained monolingual language models in cross lingual zero-shot word sense disambiguation (WSD) coupled with a contextualized mapping mechanism. We also report rigorous experi- ments that illustrate the effectiveness of em- ploying sparse contextualized word representa- tions obtained via a dictionary learning proce- dure. Our experimental results demonstrate that the above modifications yield a signifi- cant improvement of nearly 6.5 points of in- crease in the average F-score (from 62.0 to 68.5) over a collection of 17 typologically diverse set of target languages. We release our source code for replicating our experi- ments at https://github.com/begab/ sparsity_makes_sense. 1 Introduction Word sense disambiguation (WSD) is a long- standing and fundamental problem of Natural Language Processing, known to be affected by the knowledge acquisition bottleneck (Gale et al., 1992). Large pre-trained neural language models are known to effectively mitigate the problems re- lated to the paucity of high quality, large-coverage sense annotated training data for WSD (Loureiro and Jorge, 2019; Loureiro et al., 2021b; inter alia). Most recently, the knowledge acquisition bottle- neck has been identified as an immense problem in the cross-lingual setting as well (Pasini, 2020). A straightforward solution for handling this problem is to apply large multilingual pre-trained language models in a zero-shot setting, however, this ap- proach has a potential limitation owing to the curse of multilinguality (Conneau et al., 2020a), i.e., the inability of such models to handle the large number of languages involved during training such models to an equally good quality. The research community replied to the limita- tions of large massively multilingual models by de- veloping language-specific monolingual language ISO Huggingface model identifier bg DeepPavlov/bert-base-bg-cs-pl-ru-cased (Arkhipov et al., 2019) ca PlanTL-GOB-ES/roberta-base-ca (Armengol-Estapé et al., 2021) da Maltehb/danish-bert-botxo de bert-base-german-cased es dccuchile/bert-base-spanish-wwm-cased (Cañete et al., 2020) et EMBEDDIA/finest-BERT (Ulˇcar and Robnik-Šikonja, 2020) eu ixa-ehu/berteus-base-cased (Agerri et al., 2020) fr camembert-base (Martin et al., 2020) gl dvilares/bertinho-gl-base-cased (Vilares et al., 2021) hr EMBEDDIA/crosloengual-bert (Ulˇcar and Robnik-Šikonja, 2020) hu SZTAKI-HLT/hubert-base-cc (Nemeskey, 2021) it Musixmatch/umberto-commoncrawl-cased-v1 ja cl-tohoku/bert-base-japanese-whole-word-masking ko snunlp/KR-BERT-char16424 nl GroNLP/bert-base-dutch-cased (de Vries et al., 2019) sl EMBEDDIA/sloberta zh bert-base-chinese Table 1: Monolingual models from the transformers library (Wolf et al., 2020) cov- ering all the (non-English) languages of the XL-WSD dataset (Pasini et al., 2021). models.1 Table 1 provides a shortlist of recently published monolingual large pre-trained language models, related to the languages involved in the cross-lingual WSD test suit, XL-WSD (Pasini et al., 2021). With the prevalence of large monolingual pre- trained models, the important research question arises if their language-specific nature can be suc- cessfully exploited during zero-shot learning. Our research provides a thorough comparison of the application of large multilingual and monolingual …

Content truncated to 500 words