Do Nuclear Submarines Have Nuclear Captains? A Challenge Dataset for Commonsense Reasoning over Adjectives and Objects

Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing, pages 6052–6058, Hong Kong, China, November 3–7, 2019. c⃝2019 Association for Computational Linguistics 6052 Do Nuclear Submarines Have Nuclear Captains? A Challenge Dataset for Commonsense Reasoning over Adjectives and Objects James Mullenbach1∗, Jonathan Gordon2∗, Nanyun Peng3, Jonathan May3 1 ASAPP Inc. 2 Department of Computer Science, Vassar College 3 Information Sciences Institute, University of Southern California jmullenbach@asapp.com, jgordon@vassar.edu npeng,jonmay@isi.edu Abstract How do adjectives project from a noun to its parts? If a motorcycle is red, are its wheels red? Is a nuclear submarine’s captain nu- clear? These questions are easy for humans to judge using our commonsense understand- ing of the world, but are difficult for com- puters. To attack this challenge, we crowd- source a set of human judgments that answer the English-language question “Given a whole described by an adjective, does the adjective also describe a given part?” We build strong baselines for this task with a classification ap- proach. Our findings indicate that, despite the recent successes of large language models on tasks aimed to assess commonsense knowl- edge, these models do not greatly outperform simple word-level models based on pre-trained word embeddings. This provides evidence that the amount of commonsense knowledge en- coded in these language models does not ex- tend far beyond that already baked into the word embeddings. Our dataset will serve as a useful testbed for future research in common- sense reasoning, especially as it relates to ad- jectives and objects. 1 Introduction We investigate the commonsense inference of the transitivity of an attribute of a whole object to its component parts. To illustrate this targeted reason- ing by example, “is a sharp knife’s handle sharp?” The ability to perform commonsense inference of this type enables a more complete understanding of the physical world and therefore may find use in a variety of tasks in pragmatics and at the inter- face of vision and language. Consider generating a story in which a slow car goes to the shop to get a new part. If the new part is a windshield, the car remains slow, whereas if the new part is an engine, *Research conducted while author was at USC/ISI the car may now be fast. This knowledge may also help a visual agent reason about unseen objects: it knows a brick house does not have a brick door without needing to see the door. The past few years have seen a raft of data sets intended to test our ability to construct models with an understanding of commonsense knowl- edge. Standout examples are the Stanford Natu- ral Language Inference (SNLI) and related Multi- Genre Natural Language Inference (MNLI) cor- pora (Bowman et al., 2015; Williams et al., 2018), the SemEval-2018 commonsense shared task (Ostermann et al., 2018), the Rochester Story Completion (ROCStories) corpus (Mostafazadeh et al., 2016), and the Situations with Adversar- ial Generations (SWAG) grounded inference cor- pus (Zellers et …

Content truncated to 500 words