Chase: A Large-Scale and Pragmatic Chinese Dataset for Cross-Database Context-Dependent Text-to-SQL

Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing, pages 2316–2331 August 1–6, 2021. ©2021 Association for Computational Linguistics 2316 CHASE: A Large-Scale and Pragmatic Chinese Dataset for Cross-Database Context-Dependent Text-to-SQL Jiaqi Guo†∗ Ziliang Si† Yu Wang† Qian Liu‡∗ Ming Fan† Jian-Guang Lou§ Zijiang Yang†¶ and Ting Liu† †Xi’an Jiaotong University, Xi’an, China ‡Beihang University, Beijing, China §Microsoft Research, Beijing, China ¶GuardStrike Inc. {jasperguo2013,szl123,uyleewang}@stu.xjtu.edu.cn qian.liu@buaa.edu.cn jlou@microsoft.com yang@guardstrike.com {mingfan,tingliu}@mail.xjtu.edu.cn Abstract The cross-database context-dependent Text-to- SQL (XDTS) problem has attracted consid- erable attention in recent years due to its wide range of potential applications. However, we identify two biases in existing datasets for XDTS: (1) a high proportion of context- independent questions and (2) a high propor- tion of easy SQL queries. These biases con- ceal the major challenges in XDTS to some extent. In this work, we present CHASE, a large-scale and pragmatic Chinese dataset for XDTS. It consists of 5,459 coherent ques- tion sequences (17,940 questions with their SQL queries annotated) over 280 databases, in which only 35% of questions are context- independent, and 28% of SQL queries are easy. We experiment on CHASE with three state-of- the-art XDTS approaches. The best approach only achieves an exact match accuracy of 40% over all questions and 16% over all question se- quences, indicating that CHASE highlights the challenging problems of XDTS. We believe that CHASE can provide fertile soil for address- ing the problems. 1 Introduction The problem of mapping a natural language utter- ance into an executable SQL query in the cross- database and context-dependent setting has at- tracted considerable attention due to its wide range of applications (Wang et al., 2020b; Zhong et al., 2020). This problem is notoriously challenging, due to the complex contextual dependencies among questions in a sequence. Consider the question se- quence in Figure 1. In order to understand the last question, one needs to figure out the ellipti- cal object of the verb “培养(have)” from the first two questions in the sequence, which is “状元 球员(first pick player)”. Questions like this are ∗Work done during an internship at Microsoft Research. 哪所大学培养了最多 MVP 球员？ Which university has the most MVP players? SELECT T2.毕业院校 FROM MVP记录 AS T1 JOIN 球员 AS T2 ON T1.球员id = T2.球员id GROUP BY T2.毕业院校 ORDER BY COUNT(DISTINCT T2.球员id) DESC LIMIT 1; SELECT T2.college FROM MVP_record AS T1 JOIN player AS T2 ON T1.player_id = T2.player_id GROUP BY T2.college ORDER BY COUNT(DISTINCT T2.player_id) DESC LIMIT 1; 状元呢？ How about the first overall pick? 居然还是肯塔基！杜克也很出名啊，它培养了多少呢？ Still Kentucky! Duke is also very famous! How many does it have? SELECT 毕业院校 FROM 球员 WHERE 是否状元 = “是” GROUP BY 毕业院校 ORDER BY COUNT(*) DESC LIMIT 1; SELECT college FROM player WHERE is_first_pick = “yes” GROUP BY college ORDER BY COUNT(*) DESC LIMIT 1; SELECT COUNT(*) FROM 球员 WHERE 是否状元 = “是” AND 毕业院校 LIKE “%杜克%”; SELECT COUNT(*) FROM player WHERE is_first_pick = “yes” AND college LIKE “%duke%”; Figure 1: A question sequence …

Content truncated to 500 words