NL-EDIT: Correcting Semantic Parse Errors through Natural Language Interaction

Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 5599–5610 June 6–11, 2021. ©2021 Association for Computational Linguistics 5599 NL-EDIT: Correcting Semantic Parse Errors through Natural Language Interaction Ahmed Elgohary⋆∗, Christopher Meek♣, Matthew Richardson♣, Adam Fourney♣, Gonzalo Ramos♣, Ahmed Hassan Awadallah♣ ⋆University of Maryland ♣Microsoft Research elgohary@cs.umd.edu {meek,mattri,adamfo,goramos,hassanam}@microsoft.com Abstract We study semantic parsing in an interactive set- ting in which users correct errors with natural language feedback. We present NL-EDIT, a model for interpreting natural language feed- back in the interaction context to generate a sequence of edits that can be applied to the ini- tial parse to correct its errors. We show that NL-EDIT can boost the accuracy of existing text-to-SQL parsers by up to 20% with only one round of correction. We analyze the limi- tations of the model and discuss directions for improvement and evaluation. The code and datasets used in this paper are publicly avail- able at http://aka.ms/NLEdit. 1 Introduction Major progress in natural language processing has been made towards fully automating challenging tasks such as question answering, translation, and summarization. On the other hand, several studies have argued that machine learning systems that can explain their own predictions (Doshi-Velez and Kim, 2017) and learn interactively from their end- users (Amershi et al., 2014) can result in better user experiences and more effective learning systems. We develop NL-EDIT—an approach that employs both explanations and interaction in the context of semantic parsing. Most existing systems frame semantic parsing as a one-shot translation from a natural language question to the corresponding logical form (e.g., SQL query) (Yu et al., 2018a; Guo et al., 2019; Wang et al., 2020, inter alia). A growing body of recent work demonstrates that semantic parsing systems can be improved by including users in the parsing loop—giving them the affordance to examine the parses, judge their correctness, and provide feedback accordingly. The feedback often comes in the form of a binary correct/incorrect ∗Most of the work was done while the first author was an intern at Microsoft Research. Semantic Parsing: What is the full name of the candidate with the most votes? Vote_ID State Candidate_ID 4235 NC 1 2355 CA 2 1894 PA 2 Candidate_ID First_Name Last_Name 1 Donald Trump 2 Joe Biden Votes Candidates Answer: Donald Explanation: 1. For each vote_id, find the number of rows in Votes 2. Find first_name with the largest value in the result of step 1 SQL: SELECT first_name FROM candidates JOIN votes ON candidates.candidate_id = votes.candidate_id GROUP BY voter_Id ORDER BY COUNT(*) DESC LIMIT 1 Correction: It should be candidate id in step 1. Also find last name in step 2. Answer: Joe Biden Explanation: 1. For each candidate_id, find the number of rows in Votes 2. Find first_name, last_name with the largest value in the result of step 1 SQL: SELECT first_name, last_name FROM candidates JOIN votes ON candidates.candidate_id = votes.candidate_id GROUP BY candidate_Id ORDER BY COUNT(*) DESC LIMIT 1 Edit: GROUP-BY: remove vote_id GROUP-BY: …

Content truncated to 500 words