Learning to Simulate Natural Language Feedback for Interactive Semantic Parsing

Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics Volume 1: Long Papers, pages 3149–3170 July 9-14, 2023 ©2023 Association for Computational Linguistics Learning to Simulate Natural Language Feedback for Interactive Semantic Parsing Hao Yan1, Saurabh Srivastava1, Yintao Tai2∗, Sida I. Wang3, Wen-tau Yih3, Ziyu Yao1 1George Mason University, 2The University of Edinburgh, 3Meta AI 1{hyan5, ssrivas6, ziyuyao}@gmu.edu, 2y.tai-6@sms.ed.ac.uk 3{sida, scottyih}@meta.com Abstract Interactive semantic parsing based on natural language (NL) feedback, where users provide feedback to correct the parser mistakes, has emerged as a more practical scenario than the traditional one-shot semantic parsing. How- ever, prior work has heavily relied on human- annotated feedback data to train the interactive semantic parser, which is prohibitively expen- sive and not scalable. In this work, we pro- pose a new task of simulating NL feedback for interactive semantic parsing. We accompany the task with a novel feedback evaluator. The evaluator is specifically designed to assess the quality of the simulated feedback, based on which we decide the best feedback simulator from our proposed variants. On a text-to-SQL dataset, we show that our feedback simulator can generate high-quality NL feedback to boost the error correction ability of a specific parser. In low-data settings, our feedback simulator can help achieve comparable error correction performance as trained using the costly, full set of human annotations.1 1 Introduction The state of NLP research has long been domi- nated by training and evaluating single-turn mod- els, which, given a task input, produce the out- put and terminate the task immediately. However, in the more practical scenario of NLP applica- tions (e.g., smart-home virtual assistance), users often anticipate multi-turn interactions, such as be- ing able to provide feedback to the model output (De Vries et al., 2020). In doing this, not only can the model obtain more information and guid- ance to improve its task performance, but it also provides human users a mechanism to intervene in the model decision-making for safety purposes. ∗Yintao Tai was a remote intern at GMU during this project. 1Our code is publicly available at https://github.com/ hyan5/Learning_to_Simulate_NL_Feedback. What are the top 3 highest support rates? (Question) Sure! I'm getting results by the following steps: Does it sound good to you? Step 1: find the consider rate in candidate table ordered descending by consider rate Step 2: only show the first 3 rows of the results (Explanation) (Initial Logical Form Prediction) (Feedback) consider rate is wrong . Take support rate in place of it . Sure! I'm getting results by the following steps: Does it sound good to you? Step 1: find the support rate in candidate table ordered descending by support rate. Step 2: only show the first 3 rows of the results. (Explanation) (Corrected Logical Form Prediction) candidate: (Schema) SELECT consider_rate FROM candidate ORDER BY consider_rate Desc LIMIT 3 SELECT support_rate FROM candidate ORDER BY support_rate DESC LIMIT 3 consider_rate support_rate candidate_ID ... It seems right. Thanks! Error Correction Question, Initial Logical Form, Explanation, Schema Simulated Feedback Corrected Logical Form …

Content truncated to 500 words