Denoising Large-Scale Image Captioning from Alt-text Data Using Content Selection Models

Proceedings of the 29th International Conference on Computational Linguistics, pages 6089–6104 October 12–17, 2022. 6089 Denoising Large-Scale Image Captioning from Alt-text Data using Content Selection Models Khyathi Raghavi Chandu♦ 1, Piyush Sharma♣ 2, Soravit Changpinyo♠, Ashish Thapliyal♠, Radu Soricut♠ ♦ Meta AI, ♣ Uber, ♠ Google Research khyathi.research@gmail.com, {schangpi, asht, rsoricut}@google.com Abstract Training large-scale image captioning (IC) models demands access to a rich and diverse set of training examples that are expensive to curate both in terms of time and man-power. Instead, using alt-text based captions gathered from the web is a far cheaper alternative for scaling with the downside being that the data is noisy. Recent modeling approaches to IC often fall short in terms of performance in leveraging these noisy datasets as compared to datasets with clean annotations. We ad- dress this problem with a simple yet effective technique of breaking down the task into two smaller, more controllable tasks – skeleton pre- diction and skeleton-based caption generation. Specifically, we show that sub-selecting con- tent words as skeletons helps in generating im- proved and denoised captions when leveraging rich yet noisy alt-text–based uncurated datasets. We also show that the predicted English skele- tons can further cross-lingually be leveraged to generate non-English captions, by present- ing experimental results in French, Italian, Ger- man, Spanish and Hindi. We also show that skeleton-based prediction allows for better con- trol of caption properties, such as length, con- tent, and gender expression, providing a han- dle to perform human-in-the-loop interpretable semi-automatic corrections. 1 Introduction In the last demi-decade, NLP fields have ven- tured into reaping the benefits of utilizing large scale raw (uncurated) data from web-crawls. This trend aligned with new uncurated image-captioning datasets like Conceptual Captions (Sharma et al., 2018). While these uncurated datasets are supe- rior in terms of size and diversity, they are inferior to well curated datasets (Lin et al., 2014; Wang et al., 2019b) in terms of noise in the captions. The 1 Work done during internship at Google Research 2 Work done while employed at Google Research En: custom posters for a wedding . Hi: यह लेख शीषर्षक के �लए छ�व है (Translation: This is the image for the article title) It: persona ha creato un nuovo libro (Translation: person created a new book) En: collection of books on display Hi: पुस्तक का चयन (Translation: selection of books) It: una raccolta di alcuni libri (Translation: a collection of some books) Baseline Image Captioning Skeleton Prediction “collection” “book” Skeleton Based Image Captioning Proposed Approach Figure 1: Overview of our approach: (1) skeleton prediction & (2) skeleton based IC; compared to conventional IC. Output captions shown in English (En), Hindi (Hi) and Italian (It). content in the alt-text for the image is often dis- torted by the intent or context in which the image is presented. For example, the ground truth alt-text caption for a house is ‘house for sale’ instead of ‘front view of a house’. This hinders the use of these large noisy datasets to the fullest …

Content truncated to 500 words