CoSQL: A Conversational Text-to-SQL Challenge Towards Cross-Domain Natural Language Interfaces to Databases

Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing, pages 1962–1979, Hong Kong, China, November 3–7, 2019. c⃝2019 Association for Computational Linguistics 1962 CoSQL: A Conversational Text-to-SQL Challenge Towards Cross-Domain Natural Language Interfaces to Databases Tao Yu† Rui Zhang† He Yang Er† Suyi Li† Eric Xue† Bo Pang† Xi Victoria Lin¶ Yi Chern Tan† Tianze Shi§ Zihan Li‡ Youxuan Jiang‡ Michihiro Yasunaga† Sungrok Shim† Tao Chen† Alexander Fabbri† Zifan Li† Luyao Chen‡ Yuwen Zhang‡ Shreya Dixit† Vincent Zhang† Caiming Xiong¶ Richard Socher¶ Walter S. Lasecki‡ Dragomir Radev† †Yale University ¶Salesforce Research ‡University of Michigan §Cornell University {tao.yu, r.zhang, dragomir.radev}@yale.edu {xilin, cxiong, rsocher}@salesforce.com Abstract We present CoSQL, a corpus for building cross-domain, general-purpose database (DB) querying dialogue systems. It consists of 30k+ turns plus 10k+ annotated SQL queries, ob- tained from a Wizard-of-Oz (WOZ) collection of 3k dialogues querying 200 complex DBs spanning 138 domains. Each dialogue simu- lates a real-world DB query scenario with a crowd worker as a user exploring the DB and a SQL expert retrieving answers with SQL, clarifying ambiguous questions, or otherwise informing of unanswerable questions. When user questions are answerable by SQL, the ex- pert describes the SQL and execution results to the user, hence maintaining a natural interac- tion flow. CoSQL introduces new challenges compared to existing task-oriented dialogue datasets: (1) the dialogue states are grounded in SQL, a domain-independent executable rep- resentation, instead of domain-specific slot- value pairs, and (2) because testing is done on unseen databases, success requires generaliz- ing to new domains. CoSQL includes three tasks: SQL-grounded dialogue state tracking, response generation from query results, and user dialogue act prediction. We evaluate a set of strong baselines for each task and show that CoSQL presents significant challenges for future research. The dataset, baselines, and leaderboard will be released at https:// yale-lily.github.io/cosql. 1 Introduction Natural language interfaces to databases (NLIDB) have been studied extensively, with a multitude of different approaches introduced over the past few decades. To this end, considerable progress has been made in querying data via natural language (NL). However, most NL query systems expect the D1 : Database about student dormitories containing 5 tables Q1 : S1 : A1 : (Result table with many entries) R1 : Q2 : S2 : A2 : (Result table with many entries) R2 : Q3 : R3 : Q4 : S4 : A4 : Fawlty Towers R4 : Q8 : R8 : SELECT T1.dorm_name FROM dorm AS T1 JOIN has_amenity AS T2 ON T1.dormid = T2.dormid JOIN dorm_amenity AS T3 ON T2.amenid = T3.amenid WHERE T3.amenity_name = ‘TV Lounge’ EXCEPT SELECT T1.dorm_name FROM dorm AS T1 JOIN has_amenity AS T2 ON T1.dormid = T2.dormid JOIN dorm_amenity AS T3 ON T2.amenid = T3.amenid WHERE T3.amenity_name = ‘Study Room’ What are the names of all the dorms? INFORM_SQL This is the list of the names of all the dorms. CONFIRM_SQL SELECT dorm_name FROM dorm Which of those dorms have a TV lounge? …

Content truncated to 500 words