RUG-1-Pegasussers at SemEval-2022 Task 3: Data Generation Methods to Improve Recognizing Appropriate Taxonomic Word Relations

Proceedings of the 16th International Workshop on Semantic Evaluation (SemEval-2022), pages 247 - 254 July 14-15, 2022 ©2022 Association for Computational Linguistics RUG-1-Pegasussers at SemEval-2022 Task 3: Data Generation Methods to Improve Recognizing Appropriate Taxonomic Word Relations Frank van den Berg∗, Gijs Danoe∗, Esther Ploeger∗, Wessel Poelman∗ Lukas Edman, Tommaso Caselli Department of Information Science University of Groningen {f.l.van.den.berg, g.danoe, e.ploeger.1, w.g.poelman}@student.rug.nl {j.l.edman, t.caselli}@rug.nl Abstract This paper describes our system created for the SemEval 2022 Task 3: Presupposed Tax- onomies - Evaluating Neural-network Seman- tics. This task is focused on correctly recog- nizing taxonomic word relations in English, French and Italian. We develop various data generation techniques that expand the orig- inally provided train set and show that all methods increase the performance of models trained on these expanded datasets. Our final system outperforms the baseline from the task organizers by achieving an average macro F1 score of 79.6 on all languages, compared to the baseline’s 67.4. 1 Introduction In this paper, we describe our system and approach for the SemEval 2022 PreTENS (Presupposed Tax- onomies: Evaluating Neural Network Semantics) shared task (Zamparelli et al., 2022).1 The aim of this task is to gain a better understanding of the ability of language models to recognize taxonomic relations between two words. We focus on subtask 1, which is a binary clas- sification task in which a system should predict whether a sentence is valid or not, depending on the taxonomic word relation in a given sentence. We formulate the following research question: What are effective data generation ap- proaches in order to improve a language model’s ability to recognize appropriate taxonomic word relations? In our attempt to answer this question, we experi- ment with multiple approaches: adding new tem- plates, adding new nouns from similar word lists, adding additional hyponyms, inverting templates and using a paraphrasing model to create sentence ∗Contributed equally 1https://sites.google.com/view/ semeval2022-pretens/home-page variations. We use the expanded training data to fine-tune a base English BERT (Devlin et al., 2019) model for the final classification task. In our approach, we incorporate all three lan- guages for this task: English, Italian and French. Instead of generating additional data for each lan- guage and training separate models, we opt to train an English model and translate the Italian and French sentences to English, before predicting the validity labels. We choose this approach in part because several of our data generation methods are not available for French or Italian. We make use of Google Translate, as this is a widely used state-of-the-art general-domain translation system. Our model, trained on the expanded dataset, scores an average F1 score across all languages of 79.6, which is an improvement over the 67.4 baseline score. We find that the best data expansion tech- nique is to combine multiple approaches, where the output of one method is the input for the next. Our ablation experiments show that our paraphrasing method improves scores the most. All code, data and other related files can be found in our GitHub repository.2 …

Content truncated to 500 words