Effectiveness of Data Augmentation and Pretraining for Improving Neural Headline Generation in Low-Resource Settings

Proceedings of the 13th Conference on Language Resources and Evaluation (LREC 2022), pages 3561–3570 Marseille, 20-25 June 2022 © European Language Resources Association (ELRA), licensed under CC-BY-NC-4.0 3561 Effectiveness of Data Augmentation and Pretraining for Improving Neural Headline Generation in Low-Resource Settings Matej Martinc1, Syrielle Montariol2, Lidia Pivovarova3, Elaine Zosa3 1Jozef Stefan Institute, 2INRIA Paris, 3University of Helsinki matej.martinc@ijs.si, syrielle.montariol@inria.fr, first.last@helsinki.fi Abstract We tackle the problem of neural headline generation in a low-resource setting, where only limited amount of data is available to train a model. We compare the ideal high-resource scenario on English with results obtained on a smaller subset of the same data and also run experiments on two small news corpora covering low-resource languages, Croatian and Estonian. Two options for headline generation in a multilingual low-resource scenario are investigated: a pretrained multilingual encoder-decoder model and a combination of two pretrained language models, one used as an encoder and the other as a decoder, connected with a cross-attention layer that needs to be trained from scratch. The results show that the first approach outperforms the second one by a large margin. We explore several data augmentation and pretraining strategies in order to improve the performance of both models and show that while we can drastically improve the second approach using these strategies, they have little to no effect on the performance of the pretrained encoder-decoder model. Finally, we propose two new measures for evaluating the performance of the models besides the classic ROUGE scores. Keywords: Natural language generation, Headline generation, Low resource languages 1. Introduction Neural approaches for natural language generation (NLG) have mushroomed during past few years. The most common idea is to employ approaches that have shown good performance in machine translation (or an- other sequence-to-sequence task) and treat the gener- ation task as a translation task between an input text and the generated output text (Wen et al., 2015; Cho et al., 2014). The most popular text generation is auto- matic summarization, and recent years have seen huge advances in automatic generation of high-quality sum- maries. The newest approaches, such as BART (Lewis et al., 2020), employ an encoder-decoder transformer architecture (Vaswani et al., 2017), which “translates” the input text into an output summary. Due to large textual resources required by these NLG systems, research on this topic mostly focused on high- resource languages such as English, since the lack of data makes the training of these approaches from scratch infeasible in some low-resource domains and languages (Gkatzia, 2016). While recently some mul- tilingual models which also cover low-resourced lan- guages (Liu et al., 2020) have been proposed, most low-resource languages still lack efficient monolingual language generation systems. Therefore, to gener- ate texts for these languages with a neural architec- ture but without large datasets and substantial computa- tional resources—required for extensive pretraining of encoder-decoder models—we are left with two options: Using a multilingual NLG system that supports the low-resource language in which we wish to generate text. The options are limited here, with the …

Content truncated to 500 words