An Imitation Game for Learning Semantic Parsers from User Interaction

Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing, pages 6883–6902, November 16–20, 2020. c⃝2020 Association for Computational Linguistics 6883 An Imitation Game for Learning Semantic Parsers from User Interaction Ziyu Yao1, Yiqi Tang1, Wen-tau Yih2, Huan Sun1, Yu Su1 {yao.470, tang.1466, sun.397, su.809}@osu.edu scottyih@fb.com 1The Ohio State University 2Facebook AI Research, Seattle Abstract Despite the widely successful applications, building a semantic parser is still a tedious pro- cess in practice with challenges from costly data annotation and privacy risks. We sug- gest an alternative, human-in-the-loop method- ology for learning semantic parsers directly from users. A semantic parser should be in- trospective of its uncertainties and prompt for user demonstrations when uncertain. In do- ing so it also gets to imitate the user behavior and continue improving itself autonomously with the hope that eventually it may become as good as the user in interpreting their questions. To combat the sparsity of demonstrations, we propose a novel annotation-efficient imitation learning algorithm, which iteratively collects new datasets by mixing demonstrated states and confident predictions and retrains the se- mantic parser in a Dataset Aggregation fash- ion (Ross et al., 2011). We provide a theoret- ical analysis of its cost bound and also empir- ically demonstrate its promising performance on the text-to-SQL problem.1 1 Introduction Semantic parsing has found tremendous applica- tions in building natural language interfaces that allow users to query data and invoke services with- out programming (Woods, 1973; Zettlemoyer and Collins, 2005; Berant et al., 2013; Yih et al., 2015; Su et al., 2017; Yu et al., 2018). The life cycle of a semantic parser typically consists of two stages: (1) bootstrapping, where we keep collecting labeled data via trained annotators and/or crowdsourcing for model training until it reaches commercial- grade performance (e.g., 95% accuracy on a sur- rogate test set), and (2) fine-tuning, where we de- ploy the system, analyze the usage, and collect and 1Code will be available at https://github.com/ sunlab-osu/MISP. Does the system need to consider any conditions about the table attribute "School/Club Team"? No. What condition does "jalen rose" imply? (1) "Player". Thank you! Query result: 1. Executed SQL query: 25 Aleksandar Radojević Serbia Barton CC (KS) Center 5 Jalen Rose United States Michigan Guard-Forward ... ... ... ... ... How many schools or teams had jalen rose? User Interaction Feedback Collection No. Player Nationality School/Club Team Position SELECT�COUNT(School/Club�Team)�WHERE�School/Club�Team�... Question = "How many schools or teams had jalen rose?" SELECT�COUNT(School/Club�Team)�WHERE�Player�... SQL query: (System Uncertainty) SELECT�COUNT(School/Club�Team)�WHERE�Player="jalen�rose" I'm confused. Please help me out! Should I consider conditions about any of the following table attributes? (1) "Player" (2) "Nationality" (3) "Position" (4) None of the above options. Figure 1: A semantic parser proactively interacts with the user in a friendly way to resolve its uncertainties. In doing so it also gets to imitate the user behavior and continue improving itself autonomously with the hope that eventually it may become as good as the user in interpreting their questions. annotate new data to address the identified prob- …

Content truncated to 500 words