Improving Latent Alignment in Text Summarization by Generalizing the Pointer Generator

Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing, pages 3762–3773, Hong Kong, China, November 3–7, 2019. c⃝2019 Association for Computational Linguistics 3762 Improving Latent Alignment in Text Summarization by Generalizing the Pointer Generator Xiaoyu Shen1,2∗, Yang Zhao3, Hui Su4 and Dietrich Klakow1 1Spoken Language Systems (LSV), Saarland University, Germany 2Max Planck Institute for Informatics, Saarland Informatics Campus, Germany 3The University of Tokyo, Japan 4Pattern Recognition Center, Wechat AI, Tencent Inc, China Abstract Pointer Generators have been the de facto standard for modern summarization systems. However, this architecture faces two major drawbacks: Firstly, the pointer is limited to copying the exact words while ignoring possi- ble inflections or abstractions, which restricts its power of capturing richer latent align- ment. Secondly, the copy mechanism results in a strong bias towards extractive generations, where most sentences are produced by sim- ply copying from the source text. In this pa- per, we address these problems by allowing the model to “edit” pointed tokens instead of always hard copying them. The editing is per- formed by transforming the pointed word vec- tor into a target space with a learned relation embedding. On three large-scale summariza- tion dataset, we show the model is able to (1) capture more latent alignment relations than exact word matches, (2) improve word align- ment accuracy, allowing for better model inter- pretation and controlling, (3) generate higher- quality summaries validated by both qualita- tive and quantitative evaluations and (4) bring more abstraction to the generated summaries.1 1 Introduction Modern state-of-the-art (SOTA) summarization models are built upon the pointer generator archi- tecture (See et al., 2017). At each decoding step, the model generates a sentinel to decide whether to sample words based on the neural attention (gen- eration mode), or directly copy from an aligned source context (point mode) (Gu et al., 2016; Mer- ity et al., 2017; Yang et al., 2017). Though outper- forming the vanilla attention models, the pointer generator only captures exact word matches. As shown in Fig. 1, for abstractive summarization, ∗Correspondence to xshen@mpi-inf.mpg.de 1The source code is available at https://github. com/chin-gyou/generalized-PG. Figure 1: Alignment visualization of our model when decoding “closes”. Posterior alignment is more ac- curate for model interpretation. In contrast, the prior alignment probability is spared to “announced” and “closure”, which can be manually controlled to gener- ate desired summaries. Decoded samples are shown when aligned to “announced” and “closure” respec- tively. Highlighted source words are those that can be directly aligned to a target token in the gold summary. there exists a large number of syntactic inflections (escalated → escalates) or semantic transforma- tions (military campaign → war), where the tar- get word also has an explicit grounding in the source context but changes its surface. In stan- dard pointer generators, these words are not cov- ered by the point mode. This largely restricts the application of the pointer generator, especially on highly abstractive dataset where only a few words …

Content truncated to 500 words