Explicit Syntactic Guidance for Neural Text Generation

Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics Volume 1: Long Papers, pages 14095–14112 July 9-14, 2023 ©2023 Association for Computational Linguistics Explicit Syntactic Guidance for Neural Text Generation Yafu Li♠♣∗, Leyang Cui♡†, Jianhao Yan♠♣ , Yongjng Yin♠♣ Wei Bi♡ , Shuming Shi♡ , Yue Zhang♣♢† ♠ Zhejiang University ♡ Tencent AI lab ♣ School of Engineering, Westlake University ♢ Institute of Advanced Technology, Westlake Institute for Advanced Study yafuly@gmail.com {leyangcui,victoriabi,shumingshi}@tencent.com {yanjianhao,yinyongjing,zhangyue}@westlake.edu.cn Abstract Most existing text generation models follow the sequence-to-sequence paradigm. Gener- ative Grammar suggests that humans gener- ate natural language texts by learning language grammar. We propose a syntax-guided gener- ation schema, which generates the sequence guided by a constituency parse tree in a top- down direction. The decoding process can be decomposed into two parts: (1) predicting the infilling texts for each constituent in the lexical- ized syntax context given the source sentence; (2) mapping and expanding each constituent to construct the next-level syntax context. Ac- cordingly, we propose a structural beam search method to find possible syntax structures hierar- chically. Experiments on paraphrase generation and machine translation show that the proposed method outperforms autoregressive baselines, while also demonstrating effectiveness in terms of interpretability, controllability, and diversity. 1 Introduction Natural language generation (NLG), such as para- phrase generation (Sun et al., 2021), text sum- marization (Lin et al., 2018), machine translation (Vaswani et al., 2017; Edunov et al., 2018), and lan- guage models (Brown et al., 2020; OpenAI, 2023), have shown remarkable progress in the past few years. Most of the highest-performing NLG models train the model based on source-target correspon- dence and conduct autoregressive inference, which achieves competitive empirical performances yet deviates from a range of desirable attributes of hu- man language generation, e.g., lack of interpretabil- ity (Alvarez-Melis and Jaakkola, 2017; He et al., 2019; Li and Yao, 2021). It has been shown that humans generate lan- guage by learning and manipulating language gram- mar (Zholkovskii and Mel’chuk, 1965; Montague, 1974), which generative grammar (Chomsky, 1965) *Work was done during the internship at Tencent AI lab. †Corresponding authors. …………………………………………………... ………................................................................……. ………………….…. ……………………………………………….....……………………….……………………. <T> <NP> <VP>. <VP>. he seems <VP>. he seems wrong. looks like he was wrong he seems to have <VP>. looks like he made <NP>. he seems to have made a mistake. looks like he made mistakes. looks like <NP> <VP>. he seems to have made mistakes. looks like he made a mistake. …………………….……….…. he seems <S>. sounds like <S>. Figure 1: Syntax-guided generation: searching the hy- potheses hierarchically throughout the syntax tree in a top-down direction, starting from the root node “<T>”. The green blocks denote the possible syntax structures at different tree depths, the blue one denotes the external modification, whereas the gray ones denote the finalized hypotheses, marking the end of search paths. considers as a finite rule set that combines words to form grammatical sentences, thereby avoiding enu- meration of surface sequences, which can signifi- cantly increase data sparsity and reducing learning efficiency (Li et …

Content truncated to 500 words