Enhancing Video Translation Context with Object Labels

Proceedings of the 20th International Conference on Spoken Language Translation (IWSLT 2023), pages 130–137 July 13-14, 2023 c⃝2023 Association for Computational Linguistics Enhancing Video Translation Context with Object Labels Jeremy Gwinnup1,2, Tim Anderson2, Brian Ore2, Eric Hansen2, Kevin Duh1 1Johns Hopkins University, 2Air Force Research Laboratory {jeremy.gwinnup.1, timothy,anderson.20, brian.ore.1, eric.hansen.5}@us.af.mil, kevinduh@cs.jhu.edu Abstract We present a simple yet efficient method to en- hance the quality of machine translation models trained on multimodal corpora by augmenting the training text with labels of detected objects in the corresponding video segments. We then test the effects of label augmentation in both baseline and two automatic speech recognition (ASR) conditions. In contrast with multimodal techniques that merge visual and textual fea- tures, our modular method is easy to imple- ment and the results are more interpretable. Comparisons are made with Transformer trans- lation architectures trained with baseline and augmented labels, showing improvements of up to +1.0 BLEU on the How2 dataset. 1 Introduction Video streams are rich sources of content and the application of machine translation to videos present open research challenges. Specifically, we are in- terested in translating the speech content present in videos, using the visual modality as auxiliary input to improve translation quality. Intuitively, vi- sual signals may help disambiguate under-specified words or correct speech recognition errors. There has been much research in speech trans- lation, which focuses on speech input, and multi- modal machine translation, which focuses on vi- sual and textual inputs; this work combines aspects of both areas. We assume a cascaded pipeline, where the speech in a video input is first passed to a speech recognition component, then the text tran- scripts together with the video frames are passed to a multimodal machine translation (MMT) system. Our contribution is a MMT system that augments text-based training data with labels obtained from a computer vision object detector (Fig. 1). In contrast to more complex multimodal fusion techniques that combine vision and translation neu- ral networks into end-to-end models, our modu- lar approach is simple to implement, requiring no toolkit changes, and allows for easier interpretation of results. On the How2 dataset (Sanabria et al., 2018), we experiment with using clean transcripts and au- tomatic speech recognition transcripts of varying quality as input to our translation systems. This tests the effectiveness of our multimodal approach in noisy conditions, beneficial in real-world use cases. Results show gains of +0.4 to +1.0 BLEU on the How2 held-out test set. src: And then you’re going to stir it so have your stirrer available. PERSON CUP BOTTLE tgt: E então você vai mexer, então tenha seu agitador disponível. Figure 1: Demonstration of augmenting source data with detected object labels to provide additional context. 2 Object Class Label Augmentation When considering the translation of instructional videos, the speaker’s narration may use ambiguous language when describing the steps to the task as the viewer may be able to infer the intent through objects or actions in the scene. If MT systems are trained on the speaker’s …

Content truncated to 500 words