Are People Located in the Places They Mention in Their Tweets? A Multimodal Approach

Proceedings of the 29th International Conference on Computational Linguistics, pages 2561–2571 October 12–17, 2022. 2561 Are People Located in the Places They Mention in Their Tweets? A Multimodal Approach Zhaomin Xiao University of North Texas zhaominxiao@my.unt.edu Eduardo Blanco University of Arizona eduardoblanco@arizona.edu Abstract This paper introduces the problem of determin- ing whether people are located in the places they mention in their tweets. In particular, we investigate the role of text and images to solve this challenging problem. We present a new corpus of tweets that contain both text and im- ages. Our analyses show that this problem is multimodal at its core: human judgments de- pend on whether annotators have access to the text, the image, or both. Experimental results show that a neural architecture that combines both modalities yields better results. We also conduct an error analysis to provide insights into why and when each modality is beneficial. 1 Introduction Twitter is a social network in which users post short messages known as tweets. While statistics vary depending on the source and publication time, official reports state that 187 million users logged in daily in the third quarter of 2020 (Twitter, 2020), and 500 million tweets were published worldwide on a daily basis in 2014—the last year the number was made public (Twitter, 2014). According to a recent report (Pew Research Center, 2019), 24% of all Americans use Twitter (45% between 18 and 24 years of age), and 46% of them use it at least once a day (26% more than once). Tweets contain not only text (including hashtags, links, emojis, etc.), but also multimedia content such as images and videos. Indeed, 42% of tweets have images (Lee, 2015), and marketing research reveals that having an image improves user engagement: 18% more click throughs, 89% more likes and 150% more retweets (Brandwatch, 2017). When it comes to noisy user-generated content and spatial information, most previous work falls under two main topics: (a) named entity recogni- tion (Baldwin et al., 2015) and disambiguation (Es- hel et al., 2017), and (b) geolocation (Han et al., 2016). The former identifies, among others, loca- Figure 1: Examples of tweets in which the author is and is not located in the place mentioned in the tweet (Phoenix (left) and Atlanta (right) respectively). tion named entities and links them to a knowledge base without specifying who is there. The latter de- termines one location per user—even if it is not ex- plicitly mentioned. For example, place of residence can be inferred, at least to a certain degree, from the locations of other users and language usage patterns. In this paper, we tackle a complimentary problem: to determine whether people are located in the places they mention in their tweets. Extracting this kind of spatial information is challenging. First, people often mention places in their tweets even though they are not located there. Second, one must often rely on nuances in both the text and images to make a decision. Con- sider the tweets in …

Content truncated to 500 words