mCPT at SemEval-2023 Task 3: Multilingual Label-Aware Contrastive Pre-Training of Transformers for Few- and Zero-shot Framing Detection

Proceedings of the The 17th International Workshop on Semantic Evaluation (SemEval-2023), pages 941–949 July 13-14, 2023 ©2023 Association for Computational Linguistics mCPT at SemEval-2023 Task 3: Multilingual Label-Aware Contrastive Pre-Training of Transformers for Few- and Zero-shot Framing Detection Markus Reiter-Haas∗ †, Alexander Ertl∗, Kevin Innerhofer, Elisabeth Lex Graz University of Technology, Institute of Interactive Systems and Data Science Sandgasse 36/III, 8010, Graz, Austria reiter-haas@tugraz.at, ertl@student.tugraz.at innerebner@student.tugraz.at, elisabeth.lex@tugraz.at Abstract This paper presents the winning system for the zero-shot Spanish framing detection task, which also achieves competitive places in eight additional languages. The challenge of the framing detection task lies in identifying a set of 14 frames when only a few or zero samples are available, i.e., a multilingual multi-label few- or zero-shot setting. Our developed solu- tion employs a pre-training procedure based on multilingual Transformers using a label-aware contrastive loss function. In addition to de- scribing the system, we perform an embedding space analysis and ablation study to demon- strate how our pre-training procedure supports framing detection to advance computational framing analysis. 1 Introduction Approaches for computational framing detection are diverse (Ali and Hassan, 2022), as the framing concept itself is often just casually defined (Ent- man, 1993). Consequently, framing detection is challenging on its own, but also suffers from a lack of sufficient data (Kwak et al., 2020), especially in multilingual settings. The SemEval 2023 Task 3 Subtask 2 (Piskorski et al., 2023) aims at predicting 14 distinct media frames (Boydstun et al., 2013) present within news articles in 9 languages. Due to label imbalances, as a result of the high dimension of the label space compared to the number of sam- ples, traditional paradigms, e.g., per-label binary classification, do not apply well to the given setting without adaptions (Tarekegn et al., 2021). We introduce mCPT, the label-aware Contrastive Pre-training of Transformers based on a multilin- gual encoder model (original team name on the leaderboard1: PolarIce). We exploit two features ∗equal contribution †corresponding author 1https://propaganda.math.unipd.it/ semeval2023task3/SemEval2023testleaderboard.html Epochs 0 60 210 few-shot multilang E n g l i s h I t a l i a n multilang zero-shot e.g. Spanish attract repell repell and attract label 1 label 2 label 3 is_class Contrastive Fine-Tuning Head Pre-Training Head Post-Training Contrastive Fine-Tuning S t a g e s phase 1 phase 2 Figure 1: Our system performs label-aware contrastive fine-tuning (top). Embeddings of samples with similar labels are attracted, while they are repelled for dissimilar labels. The two-phase procedure (bottom) interleaves contrastive fine-tuning in both the multilingual and tar- get language training. of the task: (i) multi-label information and (ii) mul- tilingual data for pre-training. First, we leverage the label information by adopt- ing a contrastive loss function, i.e., HeroCon- Loss (Zheng et al., 2022), for natural language pro- cessing that optimizes the embedding space with respect to the similarities of the label space. There- fore, samples with more similar labels occupy simi- lar regions in the embedding space, whereas mostly dissimilar samples regarding their shared labels are pushed apart (refer to …

Content truncated to 500 words