Model-based Interactive Semantic Parsing: A Unified Framework and A Text-to-SQL Case Study

Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing, pages 5447–5458, Hong Kong, China, November 3–7, 2019. c⃝2019 Association for Computational Linguistics 5447 Model-based Interactive Semantic Parsing: A Unified Framework and A Text-to-SQL Case Study Ziyu Yao1, Yu Su1, Huan Sun1, Wen-tau Yih2∗ {yao.470, su.809, sun.397}@osu.edu scottyih@fb.com 1The Ohio State University 2Facebook AI Research, Seattle Abstract As a promising paradigm, interactive semantic parsing has shown to improve both semantic parsing accuracy and user confidence in the re- sults. In this paper, we propose a new, unified formulation of the interactive semantic parsing problem, where the goal is to design a model- based intelligent agent. The agent maintains its own state as the current predicted semantic parse, decides whether and where human inter- vention is needed, and generates a clarification question in natural language. A key part of the agent is a world model: it takes a percept (ei- ther an initial question or subsequent feedback from the user) and transitions to a new state. We then propose a simple yet remarkably ef- fective instantiation of our framework, demon- strated on two text-to-SQL datasets (WikiSQL and Spider) with different state-of-the-art base semantic parsers. Compared to an existing in- teractive semantic parsing approach that treats the base parser as a black box, our approach solicits less user feedback but yields higher run-time accuracy.1 1 Introduction Natural language interfaces that allow users to query data and invoke services without program- ming have been identified as a key application of semantic parsing (Berant et al., 2013; Thomason et al., 2015; Dong and Lapata, 2016; Zhong et al., 2017; Campagna et al., 2017; Su et al., 2017). However, existing semantic parsing technologies often fall short when deployed in practice, facing several challenges: (1) user utterances can be in- herently ambiguous or vague, making it difficult to get the correct result in one shot, (2) the ac- curacy of state-of-the-art semantic parsers are still not high enough for real use, and (3) it is hard for ∗Work started while at AI2 1Code available at https://github.com/ sunlab-osu/MISP. User Intent Actuator Error Detector Action World Model State Percept Question MISP Agent Environment Figure 1: Model-based Interactive Semantic Parsing (MISP) framework. users to validate the semantic parsing results, es- pecially with mainstream neural network models that are known for the lack of interpretability. In response to these challenges, interactive se- mantic parsing has been proposed recently as a practical solution, which includes human users in the loop to resolve utterance ambiguity, boost sys- tem accuracy, and improve user confidence via human-machine collaboration (Li and Jagadish, 2014; He et al., 2016; Chaurasia and Mooney, 2017; Su et al., 2018; Gur et al., 2018; Yao et al., 2019). For example, Gur et al. (2018) built the DialSQL system to detect errors in a generated SQL query and request user selection on alter- native options via dialogues. Similarly, Chaura- sia and Mooney (2017) and Yao et al. (2019) en- …

Content truncated to 500 words