Database reasoning over text

Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing, pages 3091–3104 August 1–6, 2021. ©2021 Association for Computational Linguistics 3091 Database Reasoning Over Text James Thorne University of Cambridge jt719@cam.ac.uk Majid Yazdani Facebook AI myazdani@fb.com Marzieh Saeidi Facebook AI marzieh@fb.com Fabrizio Silvestri Sapienza University, Rome fsilvestri @diag.uniroma1.it Sebastian Riedel Facebook AI University College London sriedel@fb.com Alon Halevy Facebook AI ayh@fb.com Abstract Neural models have shown impressive perfor- mance gains in answering queries from natu- ral language text. However, existing works are unable to support database queries, such as “List/Count all female athletes who were born in 20th century”, which require reason- ing over sets of relevant facts with operations such as join, filtering and aggregation. We show that while state-of-the-art transformer models perform very well for small databases, they exhibit limitations in processing noisy data, numerical operations, and queries that aggregate facts. We propose a modular archi- tecture to answer these database-style queries over multiple spans from text and aggregating these at scale. We evaluate the architecture using WIKINLDB,1 a novel dataset for ex- ploring such queries. Our architecture scales to databases containing thousands of facts whereas contemporary models are limited by how many facts can be encoded. In direct com- parison on small databases, our approach in- creases overall answer accuracy from 85% to 90%. On larger databases, our approach re- tains its accuracy whereas transformer base- lines could not encode the context. 1 Introduction Question answering (QA) over text has made sig- nificant strides in recent years owing to the avail- ability of new datasets and models. Machines have surpassed human performance on the well-known SQUaD task (Rajpurkar et al., 2016) where mod- els extract answer spans from a short passage of text. The subsequent body of work has further con- sidered incorporating retrieval from large corpora such as Wikipedia (Dhingra et al., 2017; Joshi et al., 2017; Kwiatkowski et al., 2019) to identify relevant information, conditioning answer generation (Chen 1https://github.com/facebookresearch/ NeuralDB Facts: (8 of 500 shown) Queries: - Nicholas lives in Washington D.C. with his wife. - Sheryl is Nicholas’s wife. - Teuvo was born in 1912 in Ruskala. - Sheryl’s mother gave birth to her in 1978. - Nicholas is a doctor. - Sarah was born in Chicago in 1982. - Sarah married John in 2010. - Sarah works in a hospital in NY as a doctor. Whose spouse is a doctor? (Join) → Sheryl, John, . . . List everyone born before 1980. (Set) → Sheryl, Teuvo, . . . Who is the oldest person? (Max) → Teuvo Who is Sheryl’s mother? (Set) → NULL Figure 1: Examples of set and aggregation queries over a natural language database: a database where facts are stored in free-form text without the need for a schema. et al., 2017; Lewis et al., 2020b; Izacard and Grave, 2020). More sophisticated architectures have been proposed with incremental retrieval for multi-hop QA (Xiong et al., 2020; Das et …

Content truncated to 500 words