Attention-Focused Adversarial Training for Robust Temporal Reasoning

Proceedings of the 13th Conference on Language Resources and Evaluation (LREC 2022), pages 7352–7359 Marseille, 20-25 June 2022 © European Language Resources Association (ELRA), licensed under CC-BY-NC-4.0 7352 Attention-Focused Adversarial Training for Robust Temporal Reasoning Lis Kanashiro Pereira1, Kevin Duh2, Fei Cheng3, Masayuki Asahara4, Ichiro Kobayashi5 Ochanomizu University1,5, John Hopkins University2, Kyoto University3 National Institute for Japanese Language and Linguistics (NINJAL) and Tokyo University of Foreign Studies 4 kanashiro.pereira@ocha.ac.jp, kevinduh@cs.jhu.edu, feicheng@i.kyoto-u.ac.jp masayu-a@ninjal.ac.jp, koba@is.ocha.ac.jp Abstract We propose an enhanced adversarial training algorithm for fine-tuning transformer-based language models (i.e., RoBERTa) and apply it to the temporal reasoning task. Current adversarial training approaches for NLP add the adversarial perturbation only to the embedding layer, ignoring the other layers of the model, which might limit the generalization power of adversarial training. Instead, our algorithm searches for the best combination of layers to add the adversarial perturbation. We add the adversarial perturbation to multiple hidden states or attention representations of the model layers. Adding the perturbation to the attention representations performed best in our experiments. Our model can improve performance on several temporal reasoning benchmarks, and establishes new state-of-the-art results. Keywords: adversarial training, robustness, temporal reasoning 1. Introduction Although recent pre-trained language models such as BERT (Devlin et al., 2018) and RoBERTa (Liu et al., 2019b) have achieved great success in a wide range of natural language processing (NLP) tasks, these models may still perform poorly on temporal reasoning scenar- ios. Ribeiro et al. (2020) has shown that such models often fail to make even simple temporal distinctions, for example, to distinguish the words before and after, resulting in degraded performance. Following best practices from recent work on enhanc- ing model generalization and robustness, we propose a model that effectively leverages pre-trained representa- tions (i.e. RoBERTa) and adversarial training for ro- bust temporal reasoning. More specifically, our main contributions are: 1) we propose an enhanced adver- sarial training algorithm for fine-tuning transformer- based language models that boosts the fine-tuning per- formance of RoBERTa. Our algorithm generates and adds the perturbation to a combination of layers during adversarial training. We propose to add the adversarial perturbation to multiple hidden states or attention vec- tors of the model layers. We hypothesize this might further encourage the model to generate more stable and better adversarial examples, and improve model generalization capability. Common adversarial train- ing approaches for NLP add the perturbation only to the embedding layer, ignoring the other layers of the model (Zhu et al., 2019; Jiang et al., 2019; Liu et al., 2020a; Pereira et al., 2020); 2) we apply our model to several temporal reasoning tasks, as they often suf- fer from having limited training, and are challenging as they require temporal knowledge usually not explic- itly stated in text; 3) we improve their state-of-the-art results on challenging temporal datasets such as MC- TACO, MATRES, and Story Cloze Task. 2. Model In this paper, we focus on fine-tuning RoBERTa models (Liu et al., 2019b) in our investigation of adversarial training, as RoBERTa has proven very …

Content truncated to 500 words