UU-Tax at SemEval-2022 Task 3: Improving the generalizability of language models for taxonomy classification through data augmentation

Proceedings of the 16th International Workshop on Semantic Evaluation (SemEval-2022), pages 271 - 281 July 14-15, 2022 ©2022 Association for Computational Linguistics UU-Tax at SemEval-2022 Task 3: Improving the generalizability of language models for taxonomy classification through data augmentation Injy Sarhan1,2, Pablo Mosteiro1, and Marco Spruit1,3,4 1Department of Information and Computing Sciences, Utrecht University, The Netherlands. 2Arab Academy for Science, Technology, and Maritime Transport, Egypt. 3Department of Public Health and Primary Care, Leiden University Medical Center, The Netherlands. 4Leiden Institute of Advanced Computer Science, Leiden University, The Netherlands. i.a.a.sarhan@uu.nl , p.mosteiro@uu.nl m.r.spruit@lumc.nl Abstract This paper presents our strategy to address the SemEval-2022 Task 3 PreTENS: Presupposed Taxonomies Evaluating Neural Network Se- mantics. The goal of the task is to identify if a sentence is deemed acceptable or not, depend- ing on the taxonomic relationship that holds between a noun pair contained in the sentence. For sub-task 1—binary classification—we pro- pose an effective way to enhance the robust- ness and the generalizability of language mod- els for better classification on this downstream task. We design a two-stage fine-tuning pro- cedure on the ELECTRA language model us- ing data augmentation techniques. Rigorous experiments are carried out using multi-task learning and data-enriched fine-tuning. Exper- imental results demonstrate that our proposed model, UU-Tax, is indeed able to generalize well for our downstream task. For sub-task 2— regression—we propose a simple classifier that trains on features obtained from Universal Sen- tence Encoder (USE). In addition to describing the submitted systems, we discuss other experi- ments that employ pre-trained language models and data augmentation techniques. For both sub-tasks, we perform error analysis to further understand the behaviour of the proposed mod- els. We achieved a global F1Binary score of 91.25% in sub-task 1 and a rho score of 0.221 in sub-task 2.1 1 Introduction Predicting the semantic relationship between words in a sentence is essential for Natural Language Processing (NLP) tasks. Deep neural language models accomplish outstanding results in multiple tasks involving semantics evaluation. The question posed by the shared task Presupposed Taxonomies: Evaluating Neural Network Semantics (PreTENS) is whether neural models can detect the taxonomic relationship between nouns, especially in scenarios 1Our implementation of UU-Tax is publicly available at https://github.com/IS5882/UU-TAX. where the pattern and/or the set of nouns in the sentence is previously unseen (Zamparelli et al., 2022). Sub-task 1 is a simpler classification task, while sub-task 2 is a more complex regression task. Both sub-tasks involve datasets in English, French and Italian. For each sub-task, teams are permitted three submissions. For each submission, the score is averaged over the three languages. The highest score from the three submissions is reported. We propose a series of models based on pre- trained language models. We enhance the provided datasets using state-of-the-art data augmentation tools, and further increase the dataset size by em- ploying translations. The aim of both steps is to create slightly modified versions of the sentences, such that the model can learn alternative forms of nouns and patterns. For the classification task (sub-task 1), …

Content truncated to 500 words