Don’t paraphrase, detect! Rapid and Effective Data Collection for Semantic Parsing

Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing, pages 3810–3820, Hong Kong, China, November 3–7, 2019. c⃝2019 Association for Computational Linguistics 3810 Don’t paraphrase, detect! Rapid and Effective Data Collection for Semantic Parsing Jonathan Herzig1 Jonathan Berant1,2 1School of Computer Science, Tel-Aviv University 2Allen Institute for Artificial Intelligence {jonathan.herzig,joberant}@cs.tau.ac.il Abstract A major hurdle on the road to conversa- tional interfaces is the difficulty in collecting data that maps language utterances to logi- cal forms. One prominent approach for data collection has been to automatically gener- ate pseudo-language paired with logical forms, and paraphrase the pseudo-language to natural language through crowdsourcing (Wang et al., 2015). However, this data collection proce- dure often leads to low performance on real data, due to a mismatch between the true dis- tribution of examples and the distribution in- duced by the data collection procedure. In this paper, we thoroughly analyze two sources of mismatch in this process: the mismatch in logical form distribution and the mismatch in language distribution between the true and in- duced distributions. We quantify the effects of these mismatches, and propose a new data col- lection approach that mitigates them. Assum- ing access to unlabeled utterances from the true distribution, we combine crowdsourcing with a paraphrase model to detect correct logi- cal forms for the unlabeled utterances. On two datasets, our method leads to 70.6 accuracy on average on the true distribution, compared to 51.3 in paraphrasing-based data collection. 1 Introduction Conversing with a virtual assistant in natural lan- guage is one of the most exciting current ap- plications of semantic parsing, the task of map- ping natural language utterances to executable log- ical forms (Zelle and Mooney, 1996; Zettlemoyer and Collins, 2005; Liang et al., 2011). Semantic parsing models rely on supervised training data that pairs natural language utterances with logi- cal forms. Alas, such data does not occur natu- rally, especially in virtual assistants that are meant to support thousands of different applications and use-cases. Thus, efficient data collection is per- c1: paper whose keyphrase is parsing c2: paper whose keyphrase is parsing and that has the largest publication year c3: number of paper whose keyphrase is parsing … x: Most recent parsing papers •Please detect the correct paraphrase: number of paper whose keyphrase is parsing • paper whose keyphrase is parsing and that has the largest publication year • paper whose keyphrase is parsing • N/A x1: How many papers use SQuAD? x2: Papers in EMNLP” x3: Most recent parsing papers x4: Largest NLP conference x4: Datasets for machine translation … Grammar output 𝒞"# Unlabeled utterances 𝒳%& 2. Crowd workers annotate unlabeled examples . 𝟖𝟐 . 𝟑𝟓 . 𝟗𝟏 st(x3,c1) st(x3,c2) st(x3,c3) 1. Unlabeled utterances are scored against canonical utterances 3. train 𝒔𝒕0𝟏(2) using all aggregated annotations Figure 1: An overview of GRANNO, a method for an- notating unlabeled utterances with their logical forms. haps the most pressing problem for scalable con- versational interfaces. …

Content truncated to 500 words