Does the Order of Training Samples Matter? Improving Neural Data-to-Text Generation with Curriculum Learning

Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics, pages 727–733 April 19 - 23, 2021. ©2021 Association for Computational Linguistics 727 Does the Order of Training Samples Matter? Improving Neural Data-to-Text Generation with Curriculum Learning Ernie Chang, Hui-Syuan Yeh, Vera Demberg Dept. of Language Science and Technology, Saarland University {cychang,yehhui,vera}@coli.uni-saarland.de Abstract Recent advancements in data-to-text genera- tion largely take on the form of neural end-to- end systems. Efforts have been dedicated to improving text generation systems by chang- ing the order of training samples in a process known as curriculum learning. Past research on sequence-to-sequence learning showed that curriculum learning helps to improve both the performance and convergence speed. In this work, we delve into the same idea surround- ing the training samples consisting of struc- tured data and text pairs, where at each up- date, the curriculum framework selects train- ing samples based on the model’s competence. Specifically, we experiment with various diffi- culty metrics and put forward a soft edit dis- tance metric for ranking training samples. Our benchmarks show faster convergence speed where training time is reduced by 38.7% and performance is boosted by 4.84 BLEU. 1 Introduction Neural data-to-text generation has been the sub- ject of much recent research. The task aims at transforming source-side structured data into target- side natural language descriptions (Reiter and Dale, 2000; Barzilay and Lapata, 2005). The process typ- ically involves mini-matches which are randomly sampled with a fixed size from the training set to feed into the model at each training step. In this paper, we apply curriculum learning to this pro- cess, which was explored in neural machine trans- lation (Platanios et al., 2019; Zhou et al., 2020), and show how it can help in neural data-to-text generation. The main idea in curriculum learning is to present the training data in a specific order, starting from easy examples and moving on to more diffi- cult ones, as the learner becomes more competent. When starting out with easier instances, the risk of getting stuck in local optima early on in training is reduced, since the loss functions in neural mod- els are typically highly non-convex (Bengio et al., 2009). This learning paradigm enables flexible batch configurations by considering the material properties as well as the state of the learner. The idea brings in two potential benefits: (1) It speeds up the convergence and reduces the computational cost. (2) It boosts the model performance, without having to change the model or add data. With the release of large data-to-text datasets (e.g. Wikibio (Lebret et al., 2016), Totto (Parikh et al., 2020), E2E (Novikova et al., 2017)), neural data-to-text generation is now at a point where train- ing speed and the order of samples may begin to make a real difference. We here show the efficacy of curriculum learning with a general LSTM-based sequence-to-sequence model and define difficulty metrics that can assess the training instances, using a sucessful competence function which estimates the model capability …

Content truncated to 500 words