Enriching Transformers with Structured Tensor-Product Representations for Abstractive Summarization

Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 4780–4793 June 6–11, 2021. ©2021 Association for Computational Linguistics 4780 Enriching Transformers with Structured Tensor-Product Representations for Abstractive Summarization Yichen Jiang∗ 1, Asli Celikyilmaz2, Paul Smolensky2,3, Paul Soulos∗ 3, Sudha Rao2, Hamid Palangi2, Roland Fernandez2, Caitlin Smith∗ 3, Mohit Bansal1, Jianfeng Gao2 1UNC Chapel Hill 2Microsoft Research, Redmond 3Johns Hopkins University {yichenj, mbansal}@cs.unc.edu {aslicel,psmo,sudha.rao,hpalangi,rfernand,jfgao}@microsoft.com {psoulos1, csmit372}@jhu.edu Abstract Abstractive summarization, the task of gener- ating a concise summary of input documents, requires: (1) reasoning over the source doc- ument to determine the salient pieces of in- formation scattered across the long document, and (2) composing a cohesive text by recon- structing these salient facts into a shorter sum- mary that faithfully reflects the complex re- lations connecting these facts. In this paper, we adapt TP-TRANSFORMER (Schlag et al., 2019), an architecture that enriches the orig- inal Transformer (Vaswani et al., 2017) with the explicitly compositional Tensor Product Representation (TPR), for the task of abstrac- tive summarization. The key feature of our model is a structural bias that we introduce by encoding two separate representations for each token to represent the syntactic structure (with role vectors) and semantic content (with filler vectors) separately. The model then binds the role and filler vectors into the TPR as the layer output. We argue that the structured interme- diate representations enable the model to take better control of the contents (salient facts) and structures (the syntax that connects the facts) when generating the summary. Empirically, we show that our TP-TRANSFORMER outper- forms the Transformer and the original TP- TRANSFORMER significantly on several ab- stractive summarization datasets based on both automatic and human evaluations. On sev- eral syntactic and semantic probing tasks, we demonstrate the emergent structural informa- tion in the role vectors and improved syntactic interpretability in the TPR layer outputs.1 1 Introduction Abstractive summarization is the task of generating a shorter version of a source text without necessar- ily reusing the sentences from the original source, ∗Work partially done while at Microsoft Research. 1Code and models are available at https://github.com/jiangycTarheel/ TPT-Summ 12 Original Text (Truncated): Authorities said the incident took place on Sao Joao beach in Caparica, south-west of Lisbon. The National Maritime Authority said a middle-aged man and a young girl died after they were unable to avoid the plane. [....] Other reports said the victims had been sunbathing when the plane made its emergency landing. […] Video footage from the scene carried by local broadcasters showed a small recreational plane parked on the sand, apparently intact and surrounded by beachgoers and emergency workers. […] Reference Summary: A man and a child have been killed after a light aircraft made an emergency landing on a beach in Portugal. Figure 1: An example document and its one line summary from XSum dataset. Document content that is composed into an abstractive summary is color-coded. while preserving the meaning of its salient contents. It is a …

Content truncated to 500 words