ChaPat at SemEval-2023 Task 9: Text Intimacy Analysis using Ensembles of Multilingual Transformers

Proceedings of the The 17th International Workshop on Semantic Evaluation (SemEval-2023), pages 1304–1310 July 13-14, 2023 ©2023 Association for Computational Linguistics ChaPat at SemEval-2023 Task 9: Text Intimacy Analysis using Ensembles of Multilingual Transformers Tanmay Chavan∗ and Ved Patwardhan∗ Pune Institute of Computer Technology {chavantanmay1402, vedpat3}@gmail.com Abstract Intimacy estimation of a given text has recently gained importance due to the increase in direct interaction of NLP systems with humans. Inti- macy is an important aspect of natural language and has a substantial impact on our everyday communication. Thus the level of intimacy can provide us with deeper insights and richer semantics of conversations. In this paper, we present our work on the SemEval shared task 9 on predicting the level of intimacy for the given text. The dataset consists of tweets in ten languages, out of which only six are avail- able in the training dataset. We conduct sev- eral experiments and show that an ensemble of multilingual models along with a language- specific monolingual model has the best perfor- mance. We also evaluate other data augmenta- tion methods such as translation and present the results. Lastly, we study the results thoroughly and present some noteworthy insights into this problem. 1 Introduction Intimacy is a crucial aspect of human nature and communication. The feeling of intimacy implies a certain degree of mutual closeness, openness, and trust between the people involved. It intends to convey information beyond the general semantic meaning, to increase the effectiveness of communi- cation. Capturing this information is a pivotal part of human-like language interaction. Although the intimacy of text is largely determined by its con- tent, several subtle semantic cues can be identified across the text to gauge its intimacy. The shared task (Pei et al., 2022) (Pei et al., 2023) was to predict the intimacy of tweets in ten languages. The primary dataset involved a set of tweets in six languages (English, Spanish, Italian, Portuguese, French, and Chinese) annotated with intimacy scores ranging from one to five. In addi- tion to this data, we used the question intimacy ∗Equal contribution dataset (Pei and Jurgens, 2020) which contains 2247 English questions from Reddit as well as another 150 questions from Books, Movies, and Twitter with intimacy scores in this dataset ranging from -1 to 1. The model performance has been evaluated on the test set in the given six languages as well as an external test set with four languages absent in the training data (Hindi, Arabic, Dutch, and Korean) based on Pearson’s r as the evaluation metric. Large language models provide a scalable and optimizable framework for performing a large range of text-classification language tasks. Trans- formers have emerged as the de-facto standard for language modeling due to their ease of paralleliza- tion and expressive power as a general-purpose computer. Owing to the democratization with pre- trained transformers, we opted for an ensemble to capture distinct sets of features and capabilities. We used bert-based mono-lingual pre-trained mod- els for all six languages as well as …

Content truncated to 500 words