Explanation Graph Generation via Pre-trained Language Models: An Empirical Study with Contrastive Learning

Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics Volume 1: Long Papers, pages 1190 - 1208 May 22-27, 2022 c⃝2022 Association for Computational Linguistics Explanation Graph Generation via Pre-trained Language Models: An Empirical Study with Contrastive Learning Swarnadeep Saha Prateek Yadav Mohit Bansal UNC Chapel Hill {swarna, prateek, mbansal}@cs.unc.edu Abstract Pre-trained sequence-to-sequence language models have led to widespread success in many natural language generation tasks. How- ever, there has been relatively less work on analyzing their ability to generate structured outputs such as graphs. Unlike natural lan- guage, graphs have distinct structural and se- mantic properties in the context of a down- stream NLP task, e.g., generating a graph that is connected and acyclic can be attributed to its structural constraints, while the semantics of a graph can refer to how meaningfully an edge represents the relation between two node con- cepts. In this work, we study pre-trained lan- guage models that generate explanation graphs in an end-to-end manner and analyze their abil- ity to learn the structural constraints and se- mantics of such graphs. We first show that with limited supervision, pre-trained language models often generate graphs that either vio- late these constraints or are semantically inco- herent. Since curating large amount of human- annotated graphs is expensive and tedious, we propose simple yet effective ways of graph per- turbations via node and edge edit operations that lead to structurally and semantically pos- itive and negative graphs. Next, we leverage these graphs in different contrastive learning models with Max-Margin and InfoNCE losses. Our methods lead to significant improvements in both structural and semantic accuracy of ex- planation graphs and also generalize to other similar graph generation tasks. Lastly, we show that human errors are the best negatives for contrastive learning and also that automat- ically generating more such human-like nega- tive graphs can lead to further improvements.1 1 Introduction Pre-trained sequence-to-sequence language mod- els (PLMs) like BART (Lewis et al., 2020) and 1Our code and models are publicly available at https: //github.com/swarnaHub/ExplagraphGen. Belief: Collectivism is terrible for society. Argument: Collectivism increases empathy. Stance: Counter Increases empathy Collectivism capable of causes Improve human relationship Terrible for society is not a Increases empathy Collectivism capable of Terrible for society is not a Empathy Society not part of Gold Graph T5-generated graph Structurally and Semantically Incorrect Belief: Since fast foods are greasy and fattening, banning them would control obesity. Argument: McDonalds has salads. Stance: Counter mcdonalds Fast Food part of not has context greasy and fattening Gold Graph T5-generated graph Semantically Incorrect Salads part of Fast Food greasy and fattening Salads banning them control obesity capable of has context part of causes Figure 1: Two representative examples from Expla- Graphs (Saha et al., 2021b) showing the belief, argu- ment, stance, gold explanation graph, and T5-generated explanation graph. The dashed nodes represent com- monsense nodes and the dashed edges are incorrect edges. The first generated graph is structurally incor- rect and the second graph is semantically incorrect. T5 (Raffel …

Content truncated to 500 words