Improving Compositional Generalization with Self-Training for Data-to-Text Generation

Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics Volume 1: Long Papers, pages 4205 - 4219 May 22-27, 2022 c⃝2022 Association for Computational Linguistics Improving Compositional Generalization with Self-Training for Data-to-Text Generation Sanket Vaibhav Mehta1∗ Jinfeng Rao2 Yi Tay3 Mihir Kale2 Ankur P. Parikh3 Emma Strubell1,3 1Carnegie Mellon University, 2Google, 3Google Research {svmehta, estrubel}@cs.cmu.edu {jinfeng, yitay, mihirkale, aparikh}@google.com Abstract Data-to-text generation focuses on generating fluent natural language responses from struc- tured meaning representations (MRs). Such representations are compositional and it is costly to collect responses for all possible combinations of atomic meaning schemata, thereby necessitating few-shot generalization to novel MRs. In this work, we systematically study the compositional generalization of the state-of-the-art T5 models in few-shot data-to- text tasks. We show that T5 models fail to generalize to unseen MRs, and we propose a template-based input representation that con- siderably improves the model’s generalization capability. To further improve the model’s performance, we propose an approach based on self-training using fine-tuned BLEURT for pseudo-response selection. On the commonly- used SGD and Weather benchmarks, the pro- posed self-training approach improves tree ac- curacy by 46%+ and reduces the slot error rates by 73%+ over the strong T5 baselines in few-shot settings. 1 1 Introduction Data-to-text generation (Dušek et al., 2020; Shen et al., 2020) is a critical component in today’s task- oriented dialog systems for producing fluent natu- ral language responses to users’ requests. The task takes structured meaning representations (MRs) as input for natural language text response generation. Such representations are compositional, which al- lows for the combination of atomic meaning units in various ways to express the rich semantics en- coded in languages. Recently, large pre-trained lan- guage models (LMs) have shown impressive results on many language understanding and generation ∗Work performed during an internship at Google. 1Our code and data is available at github.com/google- research/google-research/tree/master/compgen_d2t 250 500 750 1000 Few-shot train split size 20 40 60 80 Tree Accuracy Semantic repr. Naive Template guided Evaluation split Seen structures Unseen structures Figure 1: Performance comparison (tree accuracy) be- tween different few-shot splits and semantic representa- tions. T5-small undergoes a significant drop in perfor- mance on the unseen split and our template-guided rep- resentation improves generalization, reducing the gap. tasks (Howard and Ruder, 2018; Peters et al., 2018; Devlin et al., 2019; Raffel et al., 2020), however it remains unclear how well these LMs generalize compositionally to novel semantic representations. There have been many studies revealing that large LMs often memorize the patterns from train- ing data, while generalizing poorly to novel pat- terns. Compositionality in languages (Banarescu et al., 2013; Konstas et al., 2017) further aggravates such issues as the number of novel structural combi- nations exponentially increases with the number of atomic semantic units. In recent years, we have seen progress on benchmarking and measuring compositional generalization for languages (An- dreas, 2019), from perspectives including special- ized architectures (Lake, 2019; Rao et al., 2019) and learning strategies (Andreas, 2020; Akyürek et al., 2021). However, most …

Content truncated to 500 words