Language-Aware Multilingual Machine Translation with Self-Supervised Learning

Findings of the Association for Computational Linguistics: EACL 2023, pages 47–68 May 2-6, 2023 ©2023 Association for Computational Linguistics Fighting FIRe with FIRE: Assessing the Validity of Text-to-Video Retrieval Benchmarks Pedro Rodriguez,∗ Mahmoud Azab, Becka Silvert, Renato Sanchez, Linzy Labson, Hardik Shah, Seungwhan Moon Meta AI Abstract Searching troves of videos with textual descrip- tions is a core multimodal retrieval task. Owing to the lack of a purpose-built dataset for text-to- video retrieval, video captioning datasets have been re-purposed to evaluate models by (1) treating captions as positive matches to their respective videos and (2) assuming all other videos to be negatives. However, this methodol- ogy leads to a fundamental flaw during evalua- tion: since captions are marked as relevant only to their original video, many alternate videos also match the caption, which introduces false- negative caption-video pairs. We show that when these false negatives are corrected, a re- cent state-of-the-art model gains 25% recall points—a difference that threatens the validity of the benchmark itself. To diagnose and miti- gate this issue, we annotate and release 683K additional caption-video pairs. Using these, we recompute effectiveness scores for three mod- els on two standard benchmarks (MSR-VTT and MSVD). We find that (1) the recomputed met- rics are up to 25% recall points higher for the best models, (2) these benchmarks are nearing saturation for Recall@10, (3) caption length (generality) is related to the number of pos- itives, and (4) annotation costs can be miti- gated through sampling. We recommend re- tiring these benchmarks in their current form, and we make recommendations for future text- to-video retrieval benchmarks. 1 Introduction Text-to-video retrieval (TVR) is a challenging multi- modal retrieval task (Hu et al., 2011) with practical applications ranging from web search to organiz- ing media collections (Lew et al., 2006). To mea- sure TVR model improvement—despite a dearth of purpose-built TVR benchmarks—researchers created benchmarks by re-purposing video cap- tioning datasets such as MSR-VTT (Xu et al., ∗Correspondence to me@pedro.ai Test Caption: “Cartoon girl is talking” Predictions Rater Labels Correct@1 : 0% : 100% : , : Video 1 Video 2 Video 3 Model Test Videos: E.G., MSRVTT 1K : , : : , : Metrics Does “cartoon girl is talking” caption the video? Legend : Original Label : FIRE Label : Relevant : Irrelevant Evaluation Data Figure 1: MSR-VTT and MSVD have one positive video per caption (each video’s caption). Captions often match multiple videos, leading to false negatives. When mod- els rank false negatives highly, model quality is under- stated (full example in Appendix Figure 5). This leads to evaluations where reported metrics do not reflect their true value and are therefore not internally valid (§2.2.1). 2016), MSVD (Chen and Dolan, 2011), and Activi- tyNet (Heilbron et al., 2015; Krishna et al., 2017). Early work established an evaluation paradigm that treated captions as search queries over the collec- tion of captioned videos (Zhang et al., 2018; Yu et al., 2018; Gabeur et al., 2020); each caption and their corresponding video are …

Content truncated to 500 words