A Deep Learning based Framework for Image Paragraph Generation in Hindi

A Deep Learning based Framework for Image Paragraph Generation in Hindi Santosh Kumar Mishra1, Sushant Sinha 1, Sriparna Saha1, Pushpak Bhattacharyya2 Indian Institute of Technnology Patna 1 , Indian Institute of Technnology Bombay 2 {santosh 1821cs03, sushant 1901cs62, sriparna}@iitp.ac.in1, pb@cse.iitb.ac.in2 Abstract Generating a paragraph from an image is a complex task that requires object and activ- ity recognition. It is now feasible because of recent advances in image captioning. Sum- marizing an image into a single sentence can only provide a superficial description of the visual information included within the image. This problem can be solved by generating a detailed and coherent description of the in- put image. Most existing works on image-to- paragraph generation have been accomplished in English. We propose a novel way of gen- erating a paragraph from an image in Hindi. The world’s third most spoken language and one of India’s official languages, Hindi, is ex- tensively spoken throughout India and South Asia. We construct a new dataset for an image-to-paragraph generation in Hindi. We employ a hierarchical recurrent neural net- work (HRNN) for language modeling and an object detection model to decompose the in- put image by distinguishing regions of inter- est and objects. The performance of the pro- posed methodology is compared with other baselines in terms of BLEU, CIDER, and ME- TEOR scores, and the obtained results show that the proposed method outperforms them. 1 Introduction Human perception is dependent on vision and lan- guage. These are the most effective means of in- teracting with others. In our everyday lives, we are exposed to a large number of images from various sources such as advertisements, news stories, and the internet, among others. As a result, a technique to define the visual information and activity inside an image is essential. A major challenge in com- puter vision and natural language processing is de- veloping a system that can explain visual objects and their relationships in natural language. A computer- generated image description may aid visually im- paired people in comprehending online information (MacLeod et al., 2017). Recent advances in artificial intelligence, computer vision, and natural language processing have enabled image or video captioning to explain the visual content of an image or video (Vinyals et al., 2015) (Vinyals et al., 2016) (Ander- son et al., 2018), (Anderson et al., 2018) Because of the availability of large datasets (Young et al., 2014) (Lin et al., 2014) that combine images with natural language descriptions, it is now possible to generate sentences to interpret the images. Though the effec- tiveness of these solutions is promising, they all have one big flaw: they all fail to capture the subtleties in- side an image or video. The generated caption in the image or video cap- tioning procedure is a single sentence of around 20 words that anticipates just the essential observations within an image. Capturing all of the visual informa- tion and semantics included inside an image or video is inadequate. This problem can be solved by gener- ating …

Content truncated to 500 words