PICARD: Parsing Incrementally for Constrained Auto-Regressive Decoding from Language Models

Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 9895–9901 November 7–11, 2021. c⃝2021 Association for Computational Linguistics 9895 PICARD: Parsing Incrementally for Constrained Auto-Regressive Decoding from Language Models Torsten Scholak and Nathan Schucher and Dzmitry Bahdanau ElementAI, a ServiceNow company {torsten.scholak,dzmitry.bahdanau}@servicenow.com Abstract Large pre-trained language models for textual data have an unconstrained output space; at each decoding step, they can produce any of 10,000s of sub-word tokens. When fine-tuned to target constrained formal languages like SQL, these models often generate invalid code, rendering it unusable. We propose PICARD1, a method for constraining auto-regressive de- coders of language models through incremen- tal parsing. PICARD helps to find valid output sequences by rejecting inadmissible tokens at each decoding step. On the challenging Spider and CoSQL text-to-SQL translation tasks, we show that PICARD transforms fine-tuned T5 models with passable performance into state- of-the-art solutions. 1 Introduction While there have been many successes in applying large pre-trained language models to downstream tasks, our ability to control and constrain the out- put of these models is still very limited. Many enterprise applications are out of reach because they require a degree of rigour and exactitude that language models are not able to deliver yet. If the target is a formal language like SQL, then we would like the model to adhere exactly and provably to the SQL specification with all its lexical, grammatical, logical, and semantical constraints. Unfortunately, with pre-training alone, language models may not satisfy these correctness requirements. For text-to-SQL translation, the most widespread solution to constrained decoding is to make invalid SQL unrepresentable. For a while now it has been possible to restrict auto-regressive decoding to only those token sequences that correctly parse to SQL abstract syntax trees (Yin and Neubig, 2018; Lin et al., 2019; Wang et al., 2020). More recently, semi-auto-regressive improvements to this parsing 1The PICARD code is available at https://github. com/ElementAI/picard. 1 2 4 8 16 beam size 0.50 0.55 0.60 0.65 0.70 0.75 exact match accuracy T5-Base T5-Large T5-3B none top-2 top-4 top-8 Figure 1: Exact-set-match accuracy of the highest- scoring prediction as a function of beam size on the Spi- der text-to-SQL development set. With PICARD turned on, token predictions had to pass PICARD checking at every decoding step. Only the top-2, -4, and -8 token predictions of each hypothesis were considered in the beam search. With PICARD turned off (none), all token predictions were considered and none were checked. The models, T5-Base, -Large, and -3B, did not have access to any database content, only to the database schemas. paradigm have been proposed (Rubin and Berant, 2021). However, while effective, these approaches have in common that they are achieved at the ex- pense of using a custom vocabulary of special con- trol tokens or a custom model architecture, or both. Unfortunately, this makes them incompatible with generic pre-trained language model decoders. A less invasive and more compatible approach is to not constrain the generation process, but instead to filter finalized beam hypotheses …

Content truncated to 500 words