Database Search vs. Information Retrieval: A Novel Method for Studying Natural Language Querying of Semi-Structured Data

Proceedings of the 12th Conference on Language Resources and Evaluation (LREC 2020), pages 1772–1779 Marseille, 11–16 May 2020 c⃝ European Language Resources Association (ELRA), licensed under CC-BY-NC 1772 Database Search vs. Information Retrieval: A Novel Method for Studying Natural Language Querying of Semi-Structured Data Stefanie Nadig, Martin Braschler1, Kurt Stockinger Zurich University of Applied Sciences Winterthur, Switzerland steffinadig@gmail.com, {Martin.Braschler, Kurt.Stockinger}@zhaw.ch Abstract The traditional approach of querying a relational database is via a formal language, namely SQL. Recent developments in the design of natural language interfaces to databases show promising results for querying either with keywords or with full natural language queries and thus render relational databases more accessible to non-tech savvy users. Such enhanced relational databases basically use a search paradigm which is commonly used in the field of information retrieval. However, the way systems are evaluated in the database and the information retrieval communities often differs due to a lack of common benchmarks. In this paper, we provide an adapted benchmark data set that is based on a test collection originally used to evaluate information retrieval systems. The data set contains 45 information needs developed on the Internet Movie Database (IMDb), including corresponding relevance assessments. By mapping this benchmark data set to a relational database schema, we enable a novel way of directly comparing database search techniques with information retrieval. To demonstrate the feasibility of our approach, we present an experimental evaluation that compares SODA, a keyword-enabled relational database system, against the Terrier information retrieval system and thus lays the foundation for a future discussion of evaluating database systems that support natural language interfaces. Keywords: database search, information retrieval, benchmark data set, query evaluation, relevance assessment 1. Introduction The proliferation of World Wide Web use has democratized information search. Traditionally, technicians had crafted complicated search commands in languages such as SQL to retrieve results from relational databases. As more and more data and information became available to the public at large, new Web search services emerged that quickly became the main navigation tools for large segments of the online population. These services, with Google Web Search as the most successful among them, were built on technology from the academic field of Information Retrieval (IR) (Baeza-Yates & Ribeiro-Neto, 2011). An important characteristic of these services is that they allow the user to formulate queries in the form of either lists of keywords or even of complete, natural language sentences, thus dispensing with the technical barrier of using formal languages - allowing access to a much larger audience. Web search services such as Google Web Search have indeed become so successful, that users today expect this kind of easy, non-formal access to data and information in many more searching scenarios. While the field of IR is concerned with the question of how to retrieve results from collections of unstructured (often textual) data, research over the last few years has also become more intensive on how to provide similar, natural language interfaces to databases operating on semi-structured or structured data. To this …

Content truncated to 500 words