FaD-VLP: Fashion Vision-and-Language Pre-training towards Unified Retrieval and Captioning

Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 10484 - 10497 December 7-11, 2022 ©2022 Association for Computational Linguistics FaD-VLP: Fashion Vision-and-Language Pre-training towards Unified Retrieval and Captioning Suvir Mirchandani Stanford University suvir@cs.stanford.edu Licheng Yu Meta AI lichengyu@meta.com Mengjiao Wang Meta AI mengjiaow@meta.com Animesh Sinha Meta AI animeshsinha@meta.com Wenwen Jiang Meta AI wenwenj@meta.com Tao Xiang Meta AI / University of Surrey txiang@meta.com Ning Zhang Meta AI ningzhang@meta.com Abstract Multimodal tasks in the fashion domain have significant potential for e-commerce, but in- volve challenging vision-and-language learning problems—e.g., retrieving a fashion item given a reference image plus text feedback from a user. Prior works on multimodal fashion tasks have either been limited by the data in indi- vidual benchmarks, or have leveraged generic vision-and-language pre-training but have not taken advantage of the characteristics of fash- ion data. Additionally, these works have mainly been restricted to multimodal understanding tasks. To address these gaps, we make two key contributions. First, we propose a novel fashion-specific pre-training framework based on weakly-supervised triplets constructed from fashion image-text pairs. We show the triplet- based tasks are an effective addition to stan- dard multimodal pre-training tasks. Second, we propose a flexible decoder-based model ar- chitecture capable of both fashion retrieval and captioning tasks. Together, our model design and pre-training approach are competitive on a diverse set of fashion tasks, including cross- modal retrieval, image retrieval with text feed- back, image captioning, relative image caption- ing, and multimodal categorization. 1 Introduction Artificial intelligence has taken the fashion industry by storm in recent years. Significant advances have been made in tasks like recommendation (McAuley et al., 2015; Deldjoo et al., 2022) and virtual try-on (Han et al., 2018; Yang et al., 2022). In addition to these primarily visual tasks, multimodal tasks are of particular interest in fashion for e-commerce applications: for example, text-to-image retrieval enables a shopper to identify a desired clothing item via a language query (Zhuge et al., 2021). Cross-Modal Retrieval Image Retrieval w/ Text Feedback Multimodal Categorization Relative Image Captioning Image Captioning Long sleeve wool-blend sweater in faded pink featuring stripes in tones of white, blue, grey, black ... Is light blue with no print + Buffed leather slip-on sneakers in black ... + Sneakers; Low Top Sneakers Long sleeve hoodie in navy blue. Drawstrings at hood. Zip closure and scoop pockets at front ... + Is mint green with a smaller design FaD-VLP Figure 1: We present FaD-VLP, a flexible architecture and pre-training method that can support retrieval-based and captioning-based tasks in the fashion domain. A key opportunity to enhance customers’ shop- ping experiences is in the development of inter- active multimodal shopping assistants, whereby a user could converse with a system to identify a desired product (Yuan and Lam, 2021; Han et al., 2022). As in Figure 1, a smart assistant is desired to perform multiple diverse tasks, e.g., cross-modal retrieval, image retrieval with text feedback, multi- modal categorization, image captioning, and rela- tive image captioning. Among them, perhaps the …

Content truncated to 500 words