Adapting Unsupervised Syntactic Parsing Methodology for Discourse Dependency Parsing

Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing, pages 525–535 August 1–6, 2021. ©2021 Association for Computational Linguistics 525 KM-BART: Knowledge Enhanced Multimodal BART for Visual Commonsense Generation Yiran Xing∗ ♠ Zai Shi∗♣ Zhao Meng∗♣ Gerhard Lakemeyer♠ Yunpu Ma♦ Roger Wattenhofer♣ ♠RWTH Aachen, Germany ♦LMU Munich, Germany ♣ETH Zurich, Switzerland {yiran.xing, gerhard}@rwth-aachen.de cognitive.yunpu@gmail.com {zaishi, zhmeng, wattenhofer}@ethz.ch Abstract We present Knowledge Enhanced Multimodal BART (KM-BART), which is a Transformer- based sequence-to-sequence model capable of reasoning about commonsense knowledge from multimodal inputs of images and texts. We adapt the generative BART architec- ture (Lewis et al., 2020) to a multimodal model with visual and textual inputs. We further de- velop novel pretraining tasks to improve the model performance on the Visual Common- sense Generation (VCG) task. In particular, our pretraining task of Knowledge-based Com- monsense Generation (KCG) boosts model performance on the VCG task by leveraging commonsense knowledge from a large lan- guage model pretrained on external common- sense knowledge graphs. To the best of our knowledge, we are the first to propose a ded- icated task for improving model performance on the VCG task. Experimental results show that our model reaches state-of-the-art perfor- mance on the VCG task (Park et al., 2020) by applying these novel pretraining tasks. 1 Introduction Early work on Vision-Language models has been largely focused on pure understanding tasks (Tan and Bansal, 2019; Lu et al., 2019). These models, although improving model performance on under- standing tasks such as Visual Question Answer- ing (Antol et al., 2015), are not capable of mul- timodal generation tasks (You et al., 2016). To ease this problem, researchers have proposed vari- ous models (Zhou et al., 2020; Li et al., 2020) for generating texts based on visual inputs. These models are mainly pretrained on general visual and language understanding tasks such as masked language modeling and masked region modeling, which enable the models to build an ∗The first three authors contribute equally to this work. alignment between visual and language features. However, only feature alignments are inadequate to enhance the model’s ability in conducting com- plex multimodal commonsense reasoning, which requires the model to understand the underlying relations and effects between objects. Commonsense reasoning was traditionally stud- ied on natural language (Rajani et al., 2019; Trinh and Le, 2018), while recent works have paid at- tention to commonsense reasoning with joint vi- sual and language inputs. For instance, Zellers et al. (2019) proposes the task of Visual Common- sense Reasoning (VCR). However, the task focuses on understanding instead of generating as it asks the model to answer multiple-choice questions. A newly introduced dataset, Visual Commonsense Generation (VCG) (Park et al., 2020), provides a more challenging task by requiring the model to generate commonsense inferences about what might happen before/after, and the present intents of characters (see Table 2 for an example). In this work, we propose to tackle the task of VCG by leveraging our Knowledge Enhanced …

Content truncated to 500 words