Finding the Evidence: Localization-aware Answer Prediction for Text Visual Question Answering

Proceedings of the 28th International Conference on Computational Linguistics, pages 3118–3131 Barcelona, Spain (Online), December 8-13, 2020 3118 Finding the Evidence: Localization-aware Answer Prediction for Text Visual Question Answering Wei Han∗ MediaTek, Singapore Wei.Han@mediatek.com Hantao Huang∗ MediaTek, Singapore Hantao.Huang@mediatek.com Tao Han MediaTek, Singapore TaoH.Han@mediatek.com Abstract Image text carries essential information to understand the scene and perform reasoning. Text- based visual question answering (text VQA) task focuses on visual questions that require reading text in images. Existing text VQA systems generate an answer by selecting from optical charac- ter recognition (OCR) texts or a fixed vocabulary. Positional information of text is underused and there is a lack of evidence for the generated answer. As such, this paper proposes a localization- aware answer prediction network (LaAP-Net) to address this challenge. Our LaAP-Net not only generates the answer to the question but also predicts a bounding box as evidence of the gen- erated answer. Moreover, a context-enriched OCR representation (COR) for multimodal fusion is proposed to facilitate the localization task. Our proposed LaAP-Net outperforms existing ap- proaches on three benchmark datasets for the text VQA task by a noticeable margin. 1 Introduction Visual Question Answering (VQA) has attracted much interest from the communities and witnessed tremendous progress. However, lacking the ability to generate answers based on texts in the image limits its applications. Recently, many new datasets (Biten et al., 2019a; Singh et al., 2019) and new methods (Singh et al., 2019; Hu et al., 2020) are proposed to tackle this challenge and refer it as text VQA. The earliest method for text VQA is LoRRA(Singh et al., 2019), which provides an optical character recognition (OCR) module for the VQA input and proposes a dynamic copy mechanism to select the answer from both fixed vocabulary and OCR words. The following work M4C(Hu et al., 2020) inspired by LoRRA, uses rich representations of OCR as input and utilizes dynamic pointer network to deal with out-of-vocabulary answers, leading to state-of-the-art performance. However, M4C simply concatenates all modalities as transformer input and does not consider the high-level interaction among modalities of text VQA. Moreover, it is unable to provide evidence for the answer since the text is not localized in the image. Another recent work (Wang et al., 2020) proposes a new dataset for evidence-based text VQA, which suggests Intersection over Union (IoU) based evaluation metric to measure the evidence. Our work follows the spirit of evidence-based text VQA. More specifically, we generate the answer text bounding box during the answer prediction process as supplementary evidence for our answer. We propose a localization-aware answer prediction module (LaAP) that integrates the predicted bounding box with our semantic representation for the final answer. Besides, we propose a multimodal fusion module with context-enriched OCR representation, which uses a novel position-guided attention to integrate context object features into OCR representation. The contributions of this paper are summarized as follows: 1) We propose a LaAP module, which pre- dicts the OCR position and integrates it with the generated answer embedding for …

Content truncated to 500 words