Late Fusion with Triplet Margin Objective for Multimodal Ideology Prediction and Analysis

Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 9720 - 9736 December 7-11, 2022 ©2022 Association for Computational Linguistics Late Fusion with Triplet Margin Objective for Multimodal Ideology Prediction and Analysis Changyuan Qiu* Winston Wu* Xinliang Frederick Zhang Lu Wang Computer Science and Engineering University of Michigan {peterqiu,wuws,xlfzhang,wangluxy}@umich.edu Abstract Prior work on ideology prediction has largely focused on single modalities, i.e., text or im- ages. In this work, we introduce the task of multimodal ideology prediction, where a model predicts binary or five-point scale ideological leanings, given a text-image pair with political content. We first collect five new large-scale datasets with English documents and images along with their ideological leanings, covering news articles from a wide range of mainstream media in US and social media posts from Red- dit and Twitter. We conduct in-depth analyses on news articles and reveal differences in im- age content and usage across the political spec- trum. Furthermore, we perform extensive ex- periments and ablation studies, demonstrating the effectiveness of targeted pretraining objec- tives on different model components. Our best- performing model, a late-fusion architecture pretrained with a triplet objective over multi- modal content, outperforms the state-of-the-art text-only model by almost 4% and a strong mul- timodal baseline with no pretraining by over 3%. 1 Introduction In an increasingly divided world rife with misin- formation and hyperpartisanship, it is important to understand the perspectives and biases of the cre- ators of media that we consume. Media bias can manifest in many ways and has been analyzed from a variety of angles: the news may favor one side of a political issue (Card et al., 2015; Mendelsohn et al., 2021), select certain events to report on (Mc- Carthy et al., 1996; Oliver and Maney, 2000; Fan et al., 2019) or even distort or misrepresent facts (Gentzkow and Shapiro, 2006; Entman, 2007). Identifying a news article’s underlying political slant is the task of ideology prediction, which has * Equal contribution. (a) (b) Figure 1: Two images from separate sources depicting Federal Judge Pauses Justice Department Effort to Dis- miss Michael Flynn Case. In (a), from New York Times, Flynn is shown with several other figures and has a pos- itive expression. In contrast, in (b), from Fox News, Flynn is the sole figure, with a negative expression. focused largely on political texts like news arti- cles and has been tackled with a variety of mod- els, including Bayesian approaches with attention (Kulkarni et al., 2018), graph neural networks (Li and Goldwasser, 2019), and LSTMs and BERT (Baly et al., 2020). However, past work focuses solely on news article’s text; news articles contain other forms of non-verbal information in which the underlying ideology may be realized. Consider Figure 1a and Figure 1b, two images from articles depicting the same news story, but by news sources with opposing ideologies (New York Times and Fox News, respectively). The underly- ing ideology of the news source may influence the choice of image: in Figure 1a, Michael …

Content truncated to 500 words