Cosmos QA: Machine Reading Comprehension with Contextual Commonsense Reasoning

Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing, pages 2391–2401, Hong Kong, China, November 3–7, 2019. c⃝2019 Association for Computational Linguistics 2391 COSMOS QA: Machine Reading Comprehension with Contextual Commonsense Reasoning Lifu Huang♠⇤, Ronan Le Bras|, Chandra Bhagavatula|, Yejin Choi|,} ♠ University of Illinois Urbana-Champaign, Champaign, IL, USA | Allen Institute for Artificial Intelligence, Seattle, WA, USA } University of Washington, Seattle, WA, USA lifuh2@illinois.edu {ronanlb, chandrab, yejinc}@allenai.org Abstract Understanding narratives requires reading be- tween the lines, which in turn, requires inter- preting the likely causes and effects of events, even when they are not mentioned explicitly. In this paper, we introduce COSMOS QA, a large-scale dataset of 35, 600 problems that require commonsense-based reading compre- hension, formulated as multiple-choice ques- tions. In stark contrast to most existing reading comprehension datasets where the questions focus on factual and literal understanding of the context paragraph, our dataset focuses on reading between the lines over a diverse col- lection of people’s everyday narratives, asking such questions as “what might be the possi- ble reason of ...?”, or “what would have hap- pened if ...” that require reasoning beyond the exact text spans in the context. To establish baseline performances on COSMOS QA, we experiment with several state-of-the-art neu- ral architectures for reading comprehension, and also propose a new architecture that im- proves over the competitive baselines. Exper- imental results demonstrate a significant gap between machine (68.4%) and human perfor- mance (94%), pointing to avenues for future research on commonsense machine compre- hension. Dataset, code and leaderboard is pub- licly available at https://wilburone. github.io/cosmos. 1 Introduction Reading comprehension requires not only under- standing what is stated explicitly in text, but also reading between the lines, i.e., understanding what is not stated yet obviously true (Norvig, 1987). For example, after reading the first paragraph in Figure 1, we can understand that the writer is not a child, yet needs someone to dress him or her every ⇤*The work has been done during the author’s internship in AI2. P2: A woman had topped herself by jumping off the roof of the hospital she had just recently been admitted to. She was there because the first or perhaps latest suicide attempt was unsuccessful. She put her clothes on, folded the hospital gown and made the bed. She walked through the unit unimpeded and took the elevator to the top floor. Q: What would have happened to the woman if the staff at the hospital were doing their job properly? A: The woman would have been stopped before she left to take the elevator to the top floor and she would have lived. B: She would have been ushered to the elevator with some company. C: She would have managed to get to the elevator quicker with some assistance. D: None of the above choices. P1: It's a very humbling experience when you need someone to dress you every morning, tie your shoes, …

Content truncated to 500 words