Benchmarking Multimodal Regex Synthesis with Complex Structures

Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 6081–6094 July 5 - 10, 2020. c⃝2020 Association for Computational Linguistics 6081 Benchmarking Multimodal Regex Synthesis with Complex Structures Xi Ye Qiaochu Chen Isil Dillig Greg Durrett Department of Computer Science The University of Texas at Austin {xiye,qchen,isil,gdurrett}@cs.utexas.edu Abstract Existing datasets for regular expression (regex) generation from natural language are limited in complexity; compared to regex tasks that users post on StackOverflow, the regexes in these datasets are simple, and the language used to describe them is not diverse. We introduce STRUCTUREDREGEX, a new regex synthesis dataset differing from prior ones in three aspects. First, to obtain structurally complex and realistic regexes, we generate the regexes using a probabilistic grammar with pre-defined macros observed from real-world StackOverflow posts. Sec- ond, to obtain linguistically diverse natural language descriptions, we show crowdworkers abstract depictions of the underlying regex and ask them to describe the pattern they see, rather than having them paraphrase synthetic language. Third, we augment each regex example with a collection of strings that are and are not matched by the ground truth regex, similar to how real users give examples. Our quantitative and qualitative analysis demonstrates the advantages of STRUCTUREDREGEX over prior datasets. Further experimental results using various multimodal synthesis techniques highlight the challenge presented by our dataset, including non-local constraints and multi-modal inputs.1 1 Introduction Regular expressions (regexes) are known for their usefulness and wide applicability, and yet they are hard to understand and write, even for many programmers (Friedl, 2006). Recent research has therefore studied how to construct regexes from natural language (NL) descriptions, leading to the emergence of NL-to-regex datasets including 1Code and data available at https://www.cs. utexas.edu/˜xiye/streg/. SepTemp concat( Seg , Delimiter , Seg , Delimiter , Seg ) rep(<num>,3) <-> rep(<num>,3) rep(<num>,4) <-> Ground Truth Regex Figure Examples 012-345-6789 341-415-0341 210-543-071 210-521-73427 positive: negative: Natural Language Description I want three hyphen-separated numbers. The first and second numbers have 3 digits while the last one has 4 digits. Figure 1: Our dataset collection process. A regex is sampled from our grammar, then we render an abstract figure and generate distinguishing positive/negative ex- amples. We present the figure and examples to crowd- workers to collect natural language descriptions. KB13 (Kushman and Barzilay, 2013) and NL- TURK (Locascio et al., 2016). However, KB13 is small in size, with only 814 NL-regex pairs with even fewer distinct regexes. Locascio et al. (2016) subsequently employed a generate-and-paraphrase procedure (Wang et al., 2015) to create the larger NL-TURK dataset. However, the regexes in this dataset are very simple, and the descriptions are short, formulaic, and not linguistically diverse be- cause of the paraphrasing annotation procedure (Herzig and Berant, 2019). As a result, even when models achieve credible performance on these datasets, they completely fail when evalu- ated on the STACKOVERFLOW dataset (Ye et al., 2019), a real-world dataset collected from users seeking help on StackOverflow. The limited size of this dataset (only 62 NL-regex pairs) makes it …

Content truncated to 500 words