Apples to Apples: Learning Semantics of Common Entities Through a Novel Comprehension Task

Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics, pages 906–916 Vancouver, Canada, July 30 - August 4, 2017. c⃝2017 Association for Computational Linguistics https://doi.org/10.18653/v1/P17-1084 Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics, pages 906–916 Vancouver, Canada, July 30 - August 4, 2017. c⃝2017 Association for Computational Linguistics https://doi.org/10.18653/v1/P17-1084 Apples to Apples: Learning Semantics of Common Entities Through a Novel Comprehension Task Omid Bakhshandeh University of Rochester omidb@cs.rochester.edu James F. Allen University of Rochester Institute for Human and Machine Cognition james@cs.rochester.edu Abstract Understanding common entities and their attributes is a primary requirement for any system that comprehends natural language. In order to enable learn- ing about common entities, we intro- duce a novel machine comprehension task, GuessTwo: given a short paragraph comparing different aspects of two real- world semantically-similar entities, a sys- tem should guess what those entities are. Accomplishing this task requires deep lan- guage understanding which enables infer- ence, connecting each comparison para- graph to different levels of knowledge about world entities and their attributes. So far we have crowdsourced a dataset of more than 14K comparison paragraphs comparing entities from a variety of cat- egories such as fruits and animals. We have designed two schemes for evaluation: open-ended, and binary-choice prediction. For benchmarking further progress in the task, we have collected a set of paragraphs as the test set on which human can accom- plish the task with an accuracy of 94.2% on open-ended prediction. We have im- plemented various models for tackling the task, ranging from semantic-driven to neu- ral models. The semantic-driven approach outperforms the neural models, however, the results indicate that the task is very challenging across the models. 1 Introduction In the past few years, there has been great progress on core NLP tasks (e.g., parsing and part of speech tagging) which has renewed interest in primary language learning tasks which require text under- standing and reasoning, such as machine compre- hension (Schoenick et al., 2016; Hermann et al., 2015; Rajpurkar et al., 2016; Mostafazadeh et al., 2016). Our question is how far have we got in learning basic concepts of the world through lan- guage comprehension. If we look at the large body of work on extracting knowledge from un- structured corpora, we will see that they often lack some very basic pieces of information. For ex- ample, let us focus on the basic concept of ap- ple, the fruit. What do the state-of-the-art sys- tems and resources know about an apple? None of the state-of-the-art knowledge bases (Speer and Havasi, 2012; Carlson et al., 2010; Fader et al., 2011) include much precise information about the fact that apples have an edible skin, vary from sweet to sour, are round, and relatively the same size of a fist. Moreover, there is no clear approach on how to extract such information, if any, from trained word embeddings. This paper focuses on how we can automatically learn about various at- tributes of such generic entities in the …

Content truncated to 500 words