QuaRTz: An Open-Domain Dataset of Qualitative Relationship Questions

Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing, pages 5941–5946, Hong Kong, China, November 3–7, 2019. c⃝2019 Association for Computational Linguistics 5941 QUARTZ: An Open-Domain Dataset of Qualitative Relationship Questions Oyvind Tafjord, Matt Gardner, Kevin Lin, Peter Clark Allen Institute for Artificial Intelligence, Seattle, WA {oyvindt,mattg,kevinl peterc}@allenai.org Abstract We introduce the first open-domain dataset, called QUARTZ, for reasoning about textual qualitative relationships. QUARTZ contains general qualitative statements, e.g., “A sun- screen with a higher SPF protects the skin longer.”, twinned with 3864 crowdsourced sit- uated questions, e.g., “Billy is wearing sun- screen with a lower SPF than Lucy. Who will be best protected from the sun?”, plus anno- tations of the properties being compared. Un- like previous datasets, the general knowledge is textual and not tied to a fixed set of relation- ships, and tests a system’s ability to compre- hend and apply textual qualitative knowledge in a novel setting. We find state-of-the-art re- sults are substantially (20%) below human per- formance, presenting an open challenge to the NLP community. 1 Introduction Understanding and applying qualitative knowl- edge is a fundamental facet of intelligence. For ex- ample, we may read that exercise improves health, and thus decide to spend more time at the gym; or that larger cars cause more pollution, and thus decide to buy a smaller car to be environmen- tally sensitive. These skills require understanding the underlying qualitative relationships, and being able to apply them in specific contexts. To promote research in this direction, we present the first open-domain dataset of qualitative relationship questions, called QUARTZ (“Qualita- tive Relationship Test set”)1. Unlike earlier work in qualitative reasoning, e.g., (Tafjord et al., 2019), the dataset is not restricted to a small, fixed set of relationships. Each question Qi (2-way multi- ple choice) is grounded in a particular situation, and is paired with a sentence Ki expressing the general qualitative knowledge needed to answer it. 1Available at http://data.allenai.org/quartz/ Q: If Mona lives in a city that begins producing a greater amount of pollutants, what happens to the air quality in that city? (A) increases (B) decreases [correct] K: More pollutants mean poorer air quality. Annotations: Q: [MORE, ”greater”, ”amount of pollutants”] → (A) [MORE, ”increases”, ”air quality”] (B) [LESS, ”decreases”, ”air quality”] K: [MORE, ”more”, ”pollutants”] ↔ [LESS, ”poorer”, ”air quality”] Figure 1: QUARTZ contains situated qualitative ques- tions, each paired with a gold background knowledge sentence and qualitative annotations. Qi and Ki are also annotated with the properties being compared (Figure 1). The property annota- tions serve as supervision for a potential semantic parsing based approach. The overall task is to an- swer the Qi given the corpus K = {Ki}. We test several state-of-the-art (BERT-based) models and find that they are still substantially (20%) below human performance. Our con- tributions are thus (1) the dataset, containing 3864 richly annotated questions plus a background corpus of 400 qualitative knowledge sentences; and (2) an analysis …

Content truncated to 500 words