Evaluation Datasets for Cross-lingual Semantic Textual Similarity

Proceedings of Recent Advances in Natural Language Processing, pages 35–43 Sep 1–3, 2021. https://doi.org/10.26615/978-954-452-072-4_005 35 ArabGlossBERT: Fine-Tuning BERT on Context-Gloss Pairs for WSD Moustafa Al-Hajj Lebanese University Beirut, Lebanon moustafa.alhajj@ul.edu.lb Mustafa Jarrar∗ Birzeit University Birzeit, Palestine mjarrar@birzeit.edu Abstract Using pre-trained transformer models such as BERT has proven to be effective in many NLP tasks. This paper presents our work to fine- tune BERT models for Arabic Word Sense Disambiguation (WSD). We treated the WSD task as a sentence-pair binary classification task. First, we constructed a dataset of la- beled Arabic context-gloss pairs (∼167k pairs) we extracted from the Arabic Ontology and the large lexicographic database available at Birzeit University. Each pair was labeled as True or False and target words in each con- text were identified and annotated. Second, we used this dataset for fine-tuning three pre- trained Arabic BERT models. Third, we exper- imented the use of different supervised signals used to emphasize target words in context. Our experiments achieved promising results (accu- racy of 84%) although we used a large set of senses in the experiment. 1 Introduction Word Sense Disambiguation (WSD) aims to de- termine which sense (i.e. meaning) a word may denote in a given context. This is a challenging task due to the semantic ambiguity of words. For example, the word “book” as a noun has ten differ- ent senses in Princeton WordNet such as “a writ- ten work or composition that has been published” and “a number of pages bound together”. WSD has been a challenging task for many years but has gained recent attention due to the advances in contextualized word embedding models such as BERT (Devlin et al., 2019), ELMo (Peters et al., 2018) and GPT-2 (Radford et al., 2018). Such language models require less labeled training data since they are initially pre-trained on large corpora using self-supervised learning. The pre-trained lan- guage models can then be fine-tuned on various downstream NLP tasks such as sentiment analysis, ∗Corresponding author. social media mining, Named-Entity Recognition, word sense disambiguation, topic classification and summarization, among others. A gloss is a short dictionary definition describing one sense of a lemma or lexical entry (Jarrar, 2006, 2005). A context is an example sentence in which the lemma or one of its inflections (i.e. the target word) appears. In this paper, we aim to fine-tune Arabic models for Arabic WSD. Given a target word in a context and a set of glosses, we will fine- tune BERT models to decide which gloss is the correct sense of the target word. To do that, we converted the WSD task into a BERT sentence-pair binary classification task similar to (Huang et al., 2019; Yap et al., 2020; Blevins and Zettlemoyer, 2020). Thus, BERT is fine-tuned on a set of context- gloss pairs, where each pair is labeled as True or False to specify whether or not the gloss is the sense of the target word. In this way, the WSD task is converted into a sentence-pair classification task. One of …

Content truncated to 500 words