Unified Interpretation of Softmax Cross-Entropy and Negative Sampling: With Case Study for Knowledge Graph Embedding

Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing, pages 503–513 August 1–6, 2021. ©2021 Association for Computational Linguistics 503 E2E-VLP: End-to-End Vision-Language Pre-training Enhanced by Visual Learning Haiyang Xu∗, Ming Yan, Chenliang Li, Bin Bi, Songfang Huang, Wenming Xiao, Fei Huang Alibaba Group {shuofeng.xhy, ym119608, lcl193798, b.bi}@alibaba-inc.com {songfang.hsf, wenming.xiaowm, f.huang}@alibaba-inc.com Abstract Vision-language pre-training (VLP) on large- scale image-text pairs has achieved huge suc- cess for the cross-modal downstream tasks. The most existing pre-training methods mainly adopt a two-step training procedure, which firstly employs a pre-trained object detector to extract region-based visual features, then concatenates the image representation and text embedding as the input of Transformer to train. However, these methods face problems of using task-specific visual representation of the specific object detector for generic cross- modal understanding, and the computation in- efficiency of two-stage pipeline. In this paper, we propose the first end-to-end vision-language pre-trained model for both V+L understanding and generation, namely E2E-VLP, where we build a unified Trans- former framework to jointly learn visual rep- resentation, and semantic alignments between image and text. We incorporate the tasks of object detection and image captioning into pre- training with a unified Transformer encoder- decoder architecture for enhancing visual learning. An extensive set of experiments have been conducted on well-established vision- language downstream tasks to demonstrate the effectiveness of this novel VLP paradigm. 1 Introduction Self-supervised pre-training has achieved great suc- cess in a wide range of natural language under- standing (Devlin et al., 2018; Liu et al., 2019; Wang et al., 2019; Lan et al., 2019) and genera- tion tasks (Song et al., 2019; Lewis et al., 2019; Bi et al., 2020). Recent studies (Li et al., 2019; Lu et al., 2019; Chen et al., 2019; Tan and Bansal, 2019; Li et al., 2020b; Yu et al., 2020) have also witnessed the progress of self-supervised pre- training on vision-and-language tasks, which learns ∗corresponding author general cross-modal representations from massive image-text pairs, and fine-tunes vision-language pre-training (VLP) models on task-specific data achieving state-of-the-art results on various down- stream V+L tasks. Most existing mainstream VLP models adopt a two-step training method, which firstly extracts semantic visual features using a pre-trained object detection model, and then combines the derived object-centric representation of the image and text embedding as the input of Transformer (Vaswani et al., 2017) for cross-modal pre-training. Despite the superior performance brought by the large-scale image-text pairs, the two-stage solution suffers from the following weaknesses: 1) the object de- tection model in the first step is trained on specific visual dataset such as Visual Genome dataset (Kr- ishna et al., 2017), and the visual representation is not optimized towards a more generic cross-modal understanding in the second step. It may suffer from an error propagation problem when the object detection model fails to recognize certain important information. 2) extracting region features with an object detection model is so time-consuming that most state-of-the-art models are directly trained and …

Content truncated to 500 words