SemEval-2023 Task 9: Multilingual Tweet Intimacy Analysis

Proceedings of the The 17th International Workshop on Semantic Evaluation (SemEval-2023), pages 2235–2246 July 13-14, 2023 ©2023 Association for Computational Linguistics SemEval-2023 Task 9: Multilingual Tweet Intimacy Analysis Jiaxin Pei ♣ Vítor Silva† Maarten Bos † Yozen Liu† Leonardo Neves†‡ David Jurgens♣ Francesco Barbieri† ♣School of Information, University of Michigan, Ann Arbor, MI, USA †Snap Inc., Santa Monica, CA, USA ‡ Grammarly, San Francisco, CA, USA ♣{pedropei, jurgens}@umich.edu †{fbarbieri, vsilvasousa, maarten, yliu2}@snap.com ‡leo.neves@grammarly.com Abstract Intimacy is an important social aspect of language. Computational modeling of intimacy in language could help many downstream applications like dialogue systems and offen- siveness detection. Despite its importance, resources and approaches on modeling textual intimacy remain rare. To address this gap, we introduce MINT, a new Multilingual intimacy analysis dataset covering 13,372 tweets in 10 languages including English, French, Spanish, Italian, Portuguese, Korean, Dutch, Chinese, Hindi, and Arabic along with SemEval 2023 Task 9: Multilingual Tweet Intimacy Analysis. Our task attracted 45 participants from around the world. While the participants are able to achieve overall good performance on lan- guages in the training set, zero-shot prediction of intimacy in unseen languages remains challenging. of the task,Here we provide an overview summaries of the common approaches, and potential future directions on modeling intimacy across languages. All the relevant resources are available at https: //sites.google.com/umich.edu/ semeval-2023-tweet-intimacy. 1 Introduction Intimacy has long been viewed as a primary di- mension of human relationships and interpersonal interactions (Maslow, 1981; Sullivan, 2013; Prager, 1995). Existing studies suggest that intimacy is an essential social component of language and can be modeled with computational methods (Pei and Jur- gens, 2020). Recognizing intimacy can also serve as an important benchmark to test the ability of computational models to understand social infor- mation (Hovy and Yang, 2021). Despite the importance of intimacy in language, resources on textual intimacy analysis remain rare. Pei and Jurgens (2020) annotated the first textual in- timacy dataset containing 2,397 English questions, collected mostly from social media posts and fic- tional dialogues. However, such question phrases are often used primarily for interrogative situations, and, as such, models trained over the dataset may not generalize well to other types of text. To further promote computational modeling of textual intimacy, we introduce a new Multilingual textual intimacy dataset (MINT). The training data in MINT covers tweets in 6 languages, including English, Spanish, French, Portuguese, Italian, and Chinese, which are languages used by over 3 billion people on Earth, in The Americas, Europe, and Asia. A total of 12,000 tweets are annotated for the six languages. To test the model generalizability under zero-shot settings, we also include small test sets for Dutch, Korean, Hindi, and Arabic (500 tweets for each), which are spoken by over 0.8 billion people around the world. We benchmarked a series of large multilingual pre-trained language models including XLM-T (Barbieri et al., 2021), XLM-R (Conneau et al., 2019), BERT (Devlin et al., 2018), DistillBERT (Sanh et al., 2019) and MiniLM (Wang et al., 2020). We found that distilled models generally …

Content truncated to 500 words