Point-of-Interest Type Prediction using Text and Images

Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 7785–7797 November 7–11, 2021. c⃝2021 Association for Computational Linguistics 7785 Point-of-Interest Type Prediction using Text and Images Danae Sánchez Villegas Nikolaos Aletras Computer Science Department, University of Sheffield, UK {dsanchezvillegas1, n.aletras}@sheffield.ac.uk Abstract Point-of-interest (POI) type prediction is the task of inferring the type of a place from where a social media post was shared. Inferring a POI’s type is useful for studies in computa- tional social science including sociolinguistics, geosemiotics, and cultural geography, and has applications in geosocial networking technolo- gies such as recommendation and visualiza- tion systems. Prior efforts in POI type pre- diction focus solely on text, without taking vi- sual information into account. However in re- ality, the variety of modalities, as well as their semiotic relationships with one another, shape communication and interactions in social me- dia. This paper presents a study on POI type prediction using multimodal information from text and images available at posting time. For that purpose, we enrich a currently available data set for POI type prediction with the im- ages that accompany the text messages. Our proposed method extracts relevant information from each modality to effectively capture in- teractions between text and image achieving a macro F1 of 47.21 across eight categories significantly outperforming the state-of-the-art method for POI type prediction based on text- only methods. Finally, we provide a detailed analysis to shed light on cross-modal interac- tions and the limitations of our best perform- ing model.1 1 Introduction A place is typically described as a physical space infused with human meaning and experiences that facilitate communication (Tuan, 1977). The mul- timodal content of social media posts (e.g. text, images, emojis) generated by users from specific places such as restaurants, shops, and parks, con- tribute to shaping a place’s identity, by offering information about feelings elicited by participating 1Code and data are available here: https://github .com/danaesavi/poi-type-prediction imagine all the people sharing all the world ∼ Next stop: NYC Figure 1: Example of text and image content of sam- ple tweets. Users share content that is relevant to their experiences and feelings in the location. in an activity or living an experience in that place (Tanasescu et al., 2013). Fig. 1 shows examples of Twitter posts consist- ing of image-text pairs, shared from two different places or Point-of-Interests (POIs). Users share content that is relevant to their experience in the lo- cation. For example, the text imagine all the people sharing all the world which is accompanied by a photograph of the Imagine Mosaic in Central Park; and the text Next stop: NYC along with a picture of descriptive items that people carry at an airport such as luggage, a camera and a takeaway coffee cup. Developing computational methods to infer the type of a POI from social media posts (Liu et al., 2012; Sánchez Villegas et al., 2020) is useful for complementing studies in computational social sci- ence including sociolinguistics, geosemiotics, and cultural geography (Kress et al., …

Content truncated to 500 words