From Image to Text in Sentiment Analysis via Regression and Deep Learning

Proceedings of Recent Advances in Natural Language Processing, pages 862–868, Varna, Bulgaria, Sep 2–4, 2019. https://doi.org/10.26615/978-954-452-056-4_100 862 From Image to Text in Sentiment Analysis via Regression and Deep Learning Daniela Onita Faculty of Mathematics and Computer Science University of Bucharest danielaonita25@gmail.com Liviu P. Dinu Faculty of Mathematics and Computer Science University of Bucharest ldinu@fmi.unibuc.ro Birlutiu Adriana Computer Science Department 1 Decembrie 1918 University of Alba Iulia adriana.birlutiu@uab.ro Abstract Images and text represent types of con- tent which are used together for conveying user emotions in online social networks. These contents are usually associated with a sentiment category. In this paper, we in- vestigate an approach for mapping images to text for three types of sentiment cate- gories: positive, neutral and negative. The mapping from images to text is performed using a Kernel Ridge Regression model. We considered two types of image fea- tures: i) RGB pixel-values features, and ii) features extracted with a deep learning ap- proach. The experimental evaluation was performed on a Twitter data set containing both text and images and the sentiment as- sociated with these. The experimental re- sults show a difference in performance for different sentiment categories, in particu- lar the mapping that we propose performs better for the positive sentiment category in comparison with the neutral and nega- tive ones. Furthermore, the experimental results show that the more complex deep learning features perform better than the RGB pixel-value features for all sentiment categories and for larger training sets. 1 Introduction A quick look at an image is sufficient for a human to say a few words related to that image. How- ever, this very easy task for humans is a very diffi- cult task for the existing computer vision systems. The majority of previous work in computer vi- sion has focused on labeling images with a fixed set of visual categories. However, even though closed vocabularies of visual concepts are a conve- nient modeling assumption, they are quite restric- tive when compared to the vast amount of rich de- scriptions and impressions that a human can com- pose. Some approaches that address the challenge of generating image descriptions have been pro- posed (Kulkarni et al., 2013; Karpathy and Fei- Fei, 2015). However, these models only rely on objective image descriptors, and do not take into account the subjectivity which appears when de- scribing an image on social networks. In this work, we want to take a step forward to- wards the goal of generating subjective descrip- tions of images that are close to the natural lan- guage that is used in social networks. Figure 1 gives a hint to the motivation of our work by show- ing several samples which were used in the experi- mental evaluation. Each sample consists of an im- age and the subjective text associated to it, and has has a sentiment associated to it: negative, neutral or positive. The goal of our work is to generate subjective descriptions of images. The main challenge to- wards this goal is in …

Content truncated to 500 words