Non-Autoregressive Neural Machine Translation: A Call for Clarity

Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 248 - 260 December 7-11, 2022 ©2022 Association for Computational Linguistics Normalized Contrastive Learning for Text-Video Retrieval Yookoon Park∗1, Mahmoud Azab2, Bo Xiong2, Seungwhan Moon2, Florian Metze2, Gourab Kundu2, and Kirmani Ahmed2 1Columbia University, New York, 10027, USA yookoon.park@columbia.edu 2Meta, Menlo Park CA 94025, USA {azab,bxiong,shanemoon,fmetze,gkfacebookny,kirmani}@fb.com Abstract Cross-modal contrastive learning has led the recent advances in multimodal retrieval with its simplicity and effectiveness. In this work, however, we reveal that cross-modal con- trastive learning suffers from incorrect normal- ization of the sum retrieval probabilities of each text or video instance. Specifically, we show that many test instances are either over- or under-represented during retrieval, signifi- cantly hurting the retrieval performance. To address this problem, we propose Normalized Contrastive Learning (NCL) which utilizes the Sinkhorn-Knopp algorithm to compute the instance-wise biases that properly normalize the sum retrieval probabilities of each instance so that every text and video instance is fairly represented during cross-modal retrieval. Em- pirical study shows that NCL brings consistent and significant gains in text-video retrieval on different model architectures, with new state- of-the-art multimodal retrieval metrics on the ActivityNet, MSVD, and MSR-VTT datasets without any architecture engineering. 1 Introduction With the advent of large-scale multimodal data and transformer-based architectures, cross-modal con- trastive learning has contributed to the recent ad- vances in multimodal representation learning (Luo et al., 2021; Radford et al., 2021). Cross-modal con- trastive learning provides a simple, yet highly ef- fective approach for learning representations from multimodal data without supervisions. In particular, CLIP (Contrastive Language-Image Pretraining) (Radford et al., 2021) learns image-text represen- tations using vision and text transformer encoders on millions of image-text data from the Web and demonstrates that the learned vision and text en- coders can perform zero-shot transfer on various vision tasks. More recently, CLIP4Clip (Luo et al., ∗ Work done during internship at Meta 2021) extends the pretrained CLIP model and fine- tunes it on text-video datasets for embedding-based text-video retrieval, achieving state-of-the-art per- formance. In embedding-based retrieval, the retrieval prob- abilities of each text or video instance are defined by their embedding similarity to their cross-modal queries. When there is one-to-one correspondence between text and video instances as in many stan- dard benchmarks, one would expect that the re- trieval probabilities of a video summed over the text queries should be normalized to 1 and vice versa, so that in overall, all text and video instances are equally represented during retrieval. However, we show that in practice, cross-modal contrastive learning suffers from significant normalization er- rors of the sum retrieval probabilities of each in- stance (Fig. 2). This suggests that many test in- stances are either over- or under-represented dur- ing retrieval, leading to high false positive and false negative rates (Fig. 3) and consequently harming the text-video retrieval performance. To address this problem, we propose Normal- ized Contrastive Learning (NCL) which computes instance-wise biases using the Sinkhorn-Knopp al- gorithm (Cuturi, 2013) and adjusts the cross-modal embedding …

Content truncated to 500 words