UZH_CLyp at SemEval-2023 Task 9: Head-First Fine-Tuning and ChatGPT Data Generation for Cross-Lingual Learning in Tweet Intimacy Prediction

Proceedings of the The 17th International Workshop on Semantic Evaluation (SemEval-2023), pages 1021–1029 July 13-14, 2023 ©2023 Association for Computational Linguistics UZH_CLyp at SemEval-2023 Task 9: Head-First Fine-Tuning and ChatGPT Data Generation for Cross-Lingual Learning in Tweet Intimacy Prediction Andrianos Michail Stefanos Konstantinou Department of Computational LinguisticsSimon Clematide University of Zurich {andrianos.michail,stefanos.konstantinou,simon.clematide}@uzh.ch Abstract This paper describes the submission of UZH_CLyp for the SemEval 2023 Task 9 “Multilingual Tweet Intimacy Analysis”. We achieved second-best results in all 10 languages according to the official Pearson’s correlation regression evaluation measure. Our cross- lingual transfer learning approach explores the benefits of using a Head-First Fine-Tuning method (HeFiT) that first updates only the re- gression head parameters and then also updates the pre-trained transformer encoder parameters at a reduced learning rate. Additionally, we study the impact of using a small set of automat- ically generated examples (in our case, from ChatGPT) for low-resource settings where no human-labeled data is available. Our study shows that HeFiT stabilizes training and consis- tently improves results for pre-trained models that lack domain adaptation to tweets. Our study also shows a noticeable performance in- crease in cross-lingual learning when synthetic data is used, confirming the usefulness of cur- rent text generation systems to improve zero- shot baseline results. Finally, we examine how possible inconsistencies in the annotated data contribute to cross-lingual interference issues. 1 Introduction Social media texts are a rich source for studying social behavior, but manual analysis of them is pro- hibitively expensive. Predictive models that accu- rately estimate regression scores, such as intimacy, emotion, and valence, enable large-scale studies of social behavior through social media text. Sparse training data in numerous languages presents another challenge to globalize such studies of social behaviors. To overcome such limitations, this shared task focuses on developing models capa- ble of accurately regressing intimacy in languages without training data. In our shared task submission, we explore a novel approach to fine-tuning multilingual lan- guage models to maximize cross-lingual learning abilities. The key contributions and findings of this study are the following: • We propose an alternative fine-tuning pro- cedure for the XLM-R model that allows for an effective adaptation to the Twitter so- cial media texts without further pre-training. Our results suggest that this approach yields slightly better performance in challenging cross-lingual learning of unseen languages. • We demonstrate the effectiveness of using a small sample of synthetic tweets generated by ChatGPT1 as training data for fine-tuning multilingual models. Our results suggest that this approach can effectively improve model performance in new languages. • We examine cross-lingual interference in our experimental framework and find that some- times more training data of a language can surprisingly impair the performance of the language and related languages in the dataset. 2 Related Work Cross-lingual learning involves the transfer of knowledge between natural languages, which can be achieved through the use of multilingual lan- guage models (MLMs). MLMs are pre-trained in multiple languages and then fine-tuned using available training data, with the aim of achieving positive …

Content truncated to 500 words