Precisely the Point: Adversarial Augmentations for Faithful and Informative Text Generation

Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 7160 - 7176 December 7-11, 2022 ©2022 Association for Computational Linguistics Precisely the Point: Adversarial Augmentations for Faithful and Informative Text Generation Wenhao Wu1∗, Wei Li2, Jiachen Liu2, Xinyan Xiao2, Sujian Li1†, Yajuan Lyu2 1Key Laboratory of Computational Linguistics, MOE, Peking University 2Baidu Inc., Beijing, China {waynewu,lisujian}@pku.edu.cn {liwei85,liujiachen,xiaoxinyan, lvyajuan}@baidu.com Abstract Though model robustness has been extensively studied in language understanding, the robust- ness of Seq2Seq generation remains understud- ied. In this paper, we conduct the first quanti- tative analysis on the robustness of pre-trained Seq2Seq models. We find that even current SOTA pre-trained Seq2Seq model (BART) is still vulnerable, which leads to significant de- generation in faithfulness and informativeness for text generation tasks. This motivated us to further propose a novel adversarial augmenta- tion framework, namely AdvSeq, for generally improving faithfulness and informativeness of Seq2Seq models via enhancing their robust- ness. AdvSeq automatically constructs two types of adversarial augmentations during train- ing, including implicit adversarial samples by perturbing word representations and explicit adversarial samples by word swapping, both of which effectively improve Seq2Seq robust- ness. Extensive experiments on three popular text generation tasks demonstrate that AdvSeq significantly improves both the faithfulness and informativeness of Seq2Seq generation under both automatic and human evaluation settings. 1 Introduction Recently, text generation has made significant progress thanks to the development of pre-trained sequence-to-sequence (Seq2Seq) models (Lewis et al., 2020; Zhang et al., 2020; Roller et al., 2021). Despite being able to generate fluent and gram- matical text, current state-of-the-art models tend to generate low-informative and unfaithful outputs that hallucinated with the given inputs (Welleck et al., 2019; Maynez et al., 2020). One major rea- son is that Seq2Seq models, generally trained with Negative Log-Likelihood (NLL), are still not robust enough to handle perturbations in the input (Kang and Hashimoto, 2020). This vulnerability can lead ∗Work is done during an internship at Baidu Inc. †Corresponding author. Original Input x: Police said the incident hap- pened near the village of Tockwith. North York- shire Police said it was believed the aeroplane had suffered engine failure. Sgt Andy Graham, who attended the crash, praised the "great pilot- ing skills resulting in no injuries". Output from x: A light aircraft has crashed into a field in Yorkshire, but no-one was injured. Adversarial Input x′: Police said the incident happened near the village of Tockwith. North Yorkshire Police said it was believed the aero- planes had suffered engine failure. Sgt Andy Gra- ham, who attended the crash, praised the "great piloting skills resulting in no accident". Output from x′: Two aeroplanes have crashed in North Yorkshire, with one of them landing safely on its roof. Table 1: A sample of the XSum dataset for text sum- marization generated by fine-tuned BART, where minor perturbations (words in blue) cause low-informative and unfaithful generation (span in red). to significant degeneration of the generalization performance for text generation tasks (Cheng et al., 2019). Table 1 provides a typical example …

Content truncated to 500 words