COMET: Commonsense Transformers for Automatic Knowledge Graph Construction

Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 4762–4779 Florence, Italy, July 28 - August 2, 2019. c⃝2019 Association for Computational Linguistics 4762 COMET : Commonsense Transformers for Automatic Knowledge Graph Construction Antoine Bosselut ♦♠ Hannah Rashkin ♦♠ Maarten Sap ♦♠ Chaitanya Malaviya ♦ Asli Celikyilmaz ♣ Yejin Choi ♦♠ ♦Allen Institute for Artificial Intelligence, Seattle, WA, USA ♠Paul G. Allen School of Computer Science & Engineering, Seattle, WA, USA ♣Microsoft Research, Redmond, WA, USA Abstract We present the first comprehensive study on automatic knowledge base construction for two prevalent commonsense knowledge graphs: ATOMIC (Sap et al., 2019) and Con- ceptNet (Speer et al., 2017). Contrary to many conventional KBs that store knowledge with canonical templates, commonsense KBs only store loosely structured open-text de- scriptions of knowledge. We posit that an important step toward automatic common- sense completion is the development of gen- erative models of commonsense knowledge, and propose COMmonsEnse Transformers (COMET ) that learn to generate rich and diverse commonsense descriptions in natural language. Despite the challenges of com- monsense modeling, our investigation reveals promising results when implicit knowledge from deep pre-trained language models is transferred to generate explicit knowledge in commonsense knowledge graphs. Empirical results demonstrate that COMET is able to generate novel knowledge that humans rate as high quality, with up to 77.5% (ATOMIC) and 91.7% (ConceptNet) precision at top 1, which approaches human performance for these re- sources. Our findings suggest that using gen- erative commonsense models for automatic commonsense KB completion could soon be a plausible alternative to extractive methods. 1 Introduction When reading text, humans make commonsense inferences that frame their understanding of the narrative being presented. For machines to achieve this capability, they must be able to acquire rele- vant and correct commonsense for an unbounded set of situations. In this work, we cast common- sense acquisition as knowledge base construction and investigate whether large-scale language mod- els can effectively learn to generate the knowledge PersonX puts their arms around PersonY loving towards PersonY to comfort PersonY caring PersonX goes to the store bring a wallet feels loved Commonsense Knowledge Bases (seen events) Automatic KB Completion xAttr xAttr xIntent oReact xNeed Unseen Events PersonX buys lunch to get food xIntent xNeed nap having a rest dozing off HasSubevent HasSubevent Going to a movie having fun UsedFor energy Causes Atomic ConceptNet Throwing a party Causes Figure 1: COMET learns from an existing knowledge base (solid lines) to be able to generate novel nodes and edges (dashed lines). necessary to automatically construct a common- sense knowledge base (KB). Automatic KB construction is a long-standing goal of artificial intelligence research due to the difficulty of achieving high concept coverage in high-precision curated KBs (Lenat, 1995; Miller, 1995). Previous work has developed models capa- ble of reading and extracting semi-structured text (Suchanek et al., 2007; Hoffart et al., 2013; Auer et al., 2007; Bollacker et al., 2008) and unstruc- tured text (Dong et al., 2014; Carlson et al., 2010; Nakashole et …

Content truncated to 500 words