Transformer-based Cascaded Multimodal Speech Translation

Transformer-based Cascaded Multimodal Speech Translation Zixiu Wu1, Ozan Caglayan1, Julia Ive2, Josiah Wang1, Lucia Specia1 1Department of Computing, Imperial College London, UK 2Department of Computer Science, University of Sheffield, UK {zixiu.wu18, o.caglayan, josiah.wang, l.specia}@imperial.ac.uk j.ive@sheffield.ac.uk Abstract This paper describes the cascaded multimodal speech trans- lation systems developed by Imperial College London for the IWSLT 2019 evaluation campaign. The architecture con- sists of an automatic speech recognition (ASR) system fol- lowed by a Transformer-based multimodal machine transla- tion (MMT) system. While the ASR component is identi- cal across the experiments, the MMT model varies in terms of the way of integrating the visual context (simple condi- tioning vs. attention), the type of visual features exploited (pooled, convolutional, action categories) and the underlying architecture. For the latter, we explore both the canonical transformer [1] and its deliberation version [2] with additive and cascade variants which differ in how they integrate the textual attention. Upon conducting extensive experiments, we found that (i) the explored visual integration schemes of- ten harm the translation performance for the transformer and additive deliberation, but considerably improve the cascade deliberation; (ii) the transformer and cascade deliberation in- tegrate the visual modality better than the additive delibera- tion, as shown by the incongruence analysis. 1. Introduction The recently introduced How2 dataset [3] has stimulated re- search around multimodal language understanding through the availability of 300h instructional videos, English subti- tles and their Portuguese translations. For example, [4] suc- cessfully demonstrates that semantically rich action-based visual features are helpful in the context of machine trans- lation (MT), especially in the presence of input noise that manifests itself as missing source words. Therefore, we hy- pothesize that a speech-to-text translation (STT) system may also benefit from the visual context, especially in the tra- ditional cascaded framework [5, 6] where noisy automatic transcripts are obtained from an automatic speech recogni- tion system (ASR) and further translated into the target lan- guage using a machine translation (MT) component. The dataset enables the design of such multimodal STT systems, since we have access to a bilingual corpora as well as the corresponding audio-visual stream. Hence, in this paper, we propose a cascaded multimodal STT with two components: (i) an English ASR system trained on the How2 dataset and (ii) a transformer-based [1] visually grounded MMT system. MMT is a relatively new research topic which is interested in leveraging auxiliary modalities such as audio or vision in order to improve translation performance [7]. MMT has proved effective in scenarios such as for disambiguation [8] or when the source sentences are corrupted [9]. So far, MMT has mostly focused on integrating visual features into neural MT (NMT) systems using visual attention through convolutional feature maps [10, 11] or visual conditioning of encoder/decoder blocks through fully-connected features [12, 13, 14, 15]. Inspired by previous research in MMT, we explore several multimodal integration schemes using action-level video fea- tures. Specifically, we experiment with visually conditioning the encoder output and adding visual attention to the decoder. We further extend …

Content truncated to 500 words