BAN-Cap: A Multi-Purpose English-Bangla Image Descriptions Dataset

Proceedings of the 13th Conference on Language Resources and Evaluation (LREC 2022), pages 6855–6865 Marseille, 20-25 June 2022 © European Language Resources Association (ELRA), licensed under CC-BY-NC-4.0 6855 BAN-Cap: A Multi-Purpose English-Bangla Image Descriptions Dataset Mohammad Faiyaz Khan⊤∗, S.M. Sadiq-Ur-Rahman Shifath⊤∗, Md Saiful Islam∗† ∗Shahjalal University of Science and Technology, Sylhet, Bangladesh, {mfaiyazkhan, sm01}@student.sust.edu †University of Alberta, Edmonton, Canada mdsaifu1@ualberta.ca Abstract As computers have become efficient at understanding visual information and transforming it into a written representation, research interest in tasks like automatic image captioning has seen a significant leap over the last few years. While most of the research attention is given to the English language in a monolingual setting, resource-constrained languages like Bangla remain out of focus, predominantly due to a lack of standard datasets. Addressing this issue, we present a new dataset BAN-Cap following the widely used Flickr8k dataset, where we collect Bangla captions of the images provided by qualified annotators. Our dataset represents a wider variety of image caption styles annotated by trained people from different backgrounds. We present a quantitative and qualitative analysis of the dataset and the baseline evaluation of the recent models in Bangla image captioning. We investigate the effect of text augmentation and demonstrate that an adaptive attention-based model combined with text augmentation using Contextualized Word Replacement (CWR) outperforms all state-of-the-art models for Bangla image captioning. We also present this dataset’s multipurpose nature, especially on machine translation for Bangla-English and English-Bangla. This dataset and all the models will be useful for further research. Keywords: Image Captioning, Natural Language Processing, Multilingual, Multimodal, Machine Translation 1. Introduction Image captioning, a variety of multimodal machine learning, is a research area that integrates and models data from different modalities. It generates humanoid descriptions of images by identifying and analysing their contents. It is more involved than other computer vision or natural language processing tasks because it requires object recognition, the inference of their re- lationships, and the generation of a meaningful and relevant interpretation using a sequence of words. It leverages the heterogeneity and the correlation of data of different modalities to achieve some original goals of artificial intelligence. It has a wide range of ap- plications. For example, an image captioning system can be used in human-computer interaction, develop a hearing-aid system for visually impaired people, per- form concept-based image indexing for information re- trieval, automate self-driving cars, and many more. Because of the availability of large-scale image- sentence pair datasets like Flickr8k (Hodosh et al., 2013), Flickr30k (Young et al., 2014), and MS COCO (Lin et al., 2014), research interest in image captioning and similar domains has seen an enormous rise in the last decade. The recent advancements in deep learn- ing have made image captioning a sought-after topic for the research community. Deep learning-based mod- els like Vinyals et al. (2015) introduced significant im- provement over the traditional machine learning-based models by following the encoder-decoder architecture, which leverages the widespread sequence generation capability of the Recurrent Neural Network (RNN) conditioned by the image. …

Content truncated to 500 words