Fluent and Low-latency Simultaneous Speech-to-Speech Translation with Self-adaptive Training

Findings of the Association for Computational Linguistics: EMNLP 2020, pages 367–376 November 16 - 20, 2020. c⃝2020 Association for Computational Linguistics 367 Open-Ended Visual Question Answering by Multi-Modal Domain Adaptation Yiming Xu1∗, Lin Chen2, Zhongwei Cheng3, Lixin Duan4, and Jiebo Luo5 1 Northwestern University 2 Wyze Labs Inc 3 Amazon 4 University of Electronic Science and Technology of China 5 University of Rochester x.yiming@outlook.com, {gggchenlin,emory.cheng, lxduan, jiebo.luo}@gmail.com Abstract We study the problem of visual question an- swering (VQA) in images by exploiting su- pervised domain adaptation, where there is a large amount of labeled data in the source do- main but only limited labeled data in the tar- get domain, with the goal to train a good tar- get model. A straightforward solution is to fine-tune a pre-trained source model by us- ing those limited labeled target data, but it usually cannot work well due to the consider- able difference between the data distributions of the source and target domains. Moreover, the availability of multiple modalities (i.e., im- ages, questions and answers) in VQA poses further challenges in modeling the transfer- ability between various modalities. In this pa- per, we address the above issues by propos- ing a novel supervised multi-modal domain adaptation method for VQA to learn joint fea- ture embeddings across different domains and modalities. Specifically, we align the data dis- tributions of the source and target domains by considering those modalities both jointly and separately. Extensive experiments on VQA 2.0 and VizWiz datasets demonstrate that our pro- posed method outperforms the existing state- of-the-art baselines for open-ended VQA in this challenging domain adaptation setting. 1 Introduction The task of visual question answering (VQA) is to build a model for answering questions given an image-question pair. Recently, it has received great attention from computer vision commu- nity (Zhou et al., 2015; Kazemi and Elqursh, 2017; Tan and Bansal, 2019; Anderson et al., 2017; Kim et al., 2018; Zhang et al., 2018; Singh et al., 2019). VQA requires techniques from both im- age recognition and natural language processing, ∗This work was done during Yiming Xu’s internship at Futurewei Technologies. and most existing works use Convolutional Neural Networks (CNNs) to extract visual features from images and Recurrent Neural Networks (RNNs) to generate textual features from questions, and then combine them to generate the final answers. However, most existing VQA datasets are cre- ated in a way that is not suitable as training data for real-world applications. For example, VQA 2.0 (Goyal et al., 2019) and Visual7W (Zhu et al., 2016), arguably two of the most popular datasets for VQA, were created by using images from MSCOCO (Lin et al., 2014) with questions asked by crowd workers. Therefore, the images are typi- cally of high quality and the questions are less con- versational. On the contrary, the recently proposed VizWiz dataset (Gurari et al., 2018) was collected from blind people taking photos and asking ques- tions about those photos. Therefore, the images in VizWiz are often of poor quality, and questions …

Content truncated to 500 words