GPT-2 Contextual Data Augmentation for Word Sense Disambiguation

GPT-2 Contextual Data Augmentation for Word Sense Disambiguation Rakia Saidi LIMTIC Laboratory, Faculty of science of Tunisia UTM University / Tunisia {saidi.rakya,fjarray}@gmail.com Fethi Jarray LIMTIC Laboratory ISI Medenine Gabes University Jeongwoo Jay Kang LIG Laboratory, Emvista Univ. Grenoble Alpes France Didier Schwab LIG Laboratory Univ. Grenoble Alpes France {didier.schwab,jeongwoo.kang}@univ-grenoble-alpes.fr Abstract Most Word-Sense Disambiguation (WSD) systems rely on machine learning approaches that require large-scale corpora for effective training. So, the quality of a WSD system degrades when trained in a low-resource lan- guage such as Arabic. To improve WSD, we design a novel data augmentation technique by properly fine-tuning GPT-2 to generate a gloss or a phrase for a selected word. The generated training data is then combined with the origi- nal dataset to train a BERT-based WSD classi- fier. Experimental results show that integrating this augmentation technique improves WSD quality for both low-resource language (Ara- bic) and high-resource languages (English). 1 Introduction Polysemous is a word with several related mean- ings. Some words, such as run or set have more than thirty different meanings. Polysemous words cause ambiguity in contexts where the meaning is different from the primary meaning of the word. For example, more than 40% of English words have more than one meaning (Nagy, 1995). In nat- ural language processing, word sense disambigua- tion (WSD) consists of identifying the intended meaning (sense) of a polysemous word in a given context. Two main approaches to WSD can be distin- guished: Knowledge-Based Approach and Ma- chine Learning-Based Approach. The former re- lies on external lexical resources, such as the well- known WordNet Knowledge Base. Machine learn- ing based approaches use sense-annotated corpora to train the WSD system. The success of ML ap- proaches in general and neural networks in par- ticular highly depends on the availability of two resources inputs: 1) a dictionary or a sense inven- tory such as WordNet to specify senses for each word from some lexicon, 2) a sense annotated cor- pus for training such as SemCor. Deep neural net- works achieved state-of-the-art performance for WSD. However, they are prone to overfitting on small datasets because they learn millions or bil- lions of parameters while building the model. To avoid overfitting, the best solution is to use more training data by augmenting existing data. WSD tasks suffer from insufficient training data or unavailability of annotated corpora. To over- come this challenge, we propose to generate new corpora or increase the size of existing ones. More specifically, we propose data augmentation tech- niques based on generative language models to improve the accuracy of learning methods. In this paper, we start by discussing data aug- mentation (DA) in general and data augmenta- tion for WSD in particular, with a focus on GPT(Generative Pre-trained Transformer) such as a generative language models. To validate our proposed method, we carried out experiments on GlossBert for English WSD and on ArabicGloss- Bert for Arabic. By conducting experiments on various different WSD tasks, we show that the proposed GPT data augmentation performs bet- …

Content truncated to 500 words