TLDR: Extreme Summarization of Scientific Documents

29627

Authors

nan

Abstract

AbstractWe introduce TLDR generation, a new form of extreme summarization, for scientific papers. TLDR generation involves high source compression and requires expert background knowledge and understanding of complex domain-specific language. To facilitate study on this task, we introduce SCITLDR, a new multi-target dataset of 5.4K TLDRs over 3.2K papers. SCITLDR contains both author-written and expert-derived TLDRs, where the latter are collected using a novel annotation protocol that produces high-quality summaries while minimizing annotation burden. We propose CATTS, a simple yet effective learning strategy for generating TLDRs that exploits titles as an auxiliary training signal. CATTS improves upon strong baselines under both automated metrics and human evaluations. Data and code are publicly available at https://github.com/allenai/scitldr.

Keywords

nan

Full Content

Findings of the Association for Computational Linguistics: EMNLP 2020, pages 4766–4777 November 16 - 20, 2020. c⃝2020 Association for Computational Linguistics 4766 TLDR:�Extreme�Summarization�of�Scientiﬁc�Documents� Isabel�Cachola† Kyle�Lo† Arman�Cohan† Daniel�S.�Weld†‡ †Allen�Institute�for�AI� ‡Paul�G.�Allen�School�of�Computer�Science�&�Engineering,�University�of�Washington� {isabelc,kylel,armanc,danw}@allenai.org Abstract� We� introduce� TLDR� generation,� a� new� form� of� extreme� summarization,� for� scientiﬁc� pa- pers.� TLDR� generation� involves� high� source� compression�and�requires�expert�background� knowledge� and� understanding� of� complex� domain-speciﬁc�language.� To�facilitate�study� on� this� task,� we� introduce� SCITLDR,� a� new� multi-target�dataset�of�5.4K�TLDRs�over�3.2K� papers.�SCITLDR�contains both author-written� and� expert-derived� TLDRs,� where� the� latter� are� collected� using� a� novel� annotation� pro- tocol� that� produces� high-quality� summaries� while�minimizing�annotation�burden.� We�pro- pose� CATTS,� a� simple� yet� effective� learn- ing� strategy� for� generating� TLDRs� that� ex- ploits� titles� as� an� auxiliary� training� signal.� CATTS� improves� upon� strong� baselines� un- der�both�automated�metrics�and�human�evalua- tions.� Data�and�code�are�publicly�available�at� https://github.com/allenai/scitldr.� 1� Introduction� We�introduce�TLDR1� generation�for�scientiﬁc�pa- pers.�An�alternative�to�abstracts,�TLDRs�focus�on� the�key�aspects�of�the�paper,�such�as�its�main�con- tributions,�eschewing�nonessential�background�or� methodological�details.�Given�the�increasing�pace� of�publication�(Van�Noorden,�2014)�and�resulting� difﬁculty�in�keeping�up�with�the�literature,�TLDRs� can�enable�readers�to�quickly�discern�a�paper’s�key� points�and�decide�whether�it’s�worth�reading.�The� goal�of�existing�work�in�summarization�of�scientiﬁc� documents�is�to�generate�abstracts�or�provide�com- plimentary�summaries�to�abstracts.�(Collins�et�al.,� 2017;�Cohan�et�al.,�2018;�Chandrasekaran�et�al.,� 2019;�Yasunaga�et�al.,�2019).� In�contrast,�TLDR� 1TLDR�is�an�acronym�that�stands�for�“too�long;�didn’t�read,”� which�is�often�used�in�online�informal�discussion�(e.g.,�Twitter� or�Reddit)�about�scientiﬁc�papers.�For�visual�clarity,�we�omit� the�semi-colon.� Abstract While many approaches to make neural networks more fathomable have been proposed, they are restricted to in- terrogating the network with input data. [...] In this work, we propose neural persistence, a complexity measure for neural net- work architectures based on topological data analysis on weighted stratiﬁed graphs. [...] Intro [...] In this work, we present the following contribu- tions: We introduce neural persistence, a novel measure for char- acterizing the structural complexity of neural networks that can be eﬃciently computed. [...] Conclusion [...] However, this did not yield an early stop- ping measure because it was never triggered, thereby suggesting that neural persistence captures salient information that would otherwise be hidden among all the weights of a network [...] TLDR We develop a new topological complexity measure for deep neural networks and demonstrate that it captures their salient properties. 1 Figure�1:� An�example� TLDR� of�a�scientiﬁc�paper.� A� TLDR�is�typically�composed�of�salient�information�(in- dicated�by�colored�spans)�found�in�the�abstract,�intro,� and�conclusion�sections�of�a�paper.� generation�seeks�to�produce�an�extreme�(single�sen- tence)�summary�(Narayan�et�al.,�2018)�given�the� entire�paper.� Further,�TLDR�generation�is�a�chal- lenging�natural�language�generation�task.�Writing� a�TLDR�of�a�scientiﬁc�paper�requires�expert�back- ground�knowledge�and�understanding�of�complex� domain-speciﬁc�language�to�identify�the�salient�as- pects�of�the�paper,�while�maintaining�faithfulness�to� the�source�and�correctness�of�the�written�summary.� An�example�TLDR�is�provided�in�Figure�1.� To�facilitate�the�study�of�TLDR�generation,�we� introduce�SCITLDR,�a�new�dataset�of�5,411�TLDRs� of�computer�science�papers.�SCITLDR�is�built�from� a�combination�of�TLDRs�written�by�authors�of�sub- missions�on�OpenReview2�and�TLDRs�derived�by�a� novel�annotation�protocol�that�asks�domain�experts� to�rewrite�peer�review�comments�for�that�submis- sion.�Having�multiple�gold�summaries�per�paper�is� especially�important�for�evaluation�when�there�is� 2https://openreview.net/� 4767 variability�in�human-written�gold�summaries�(Zech- ner,�1996;�Harman�and�Over,�2004).� In� addition� to� establishing� strong� extractive� and� abstractive� summarization� baselines� using� Transformer-based� (Vaswani� et� al.,� 2017)� mod- els,�we�present�CATTS�(Controlled�Abstraction�for� TLDRs�with�Title�Scaffolding),�a�simple�yet�effec- tive�learning�strategy�for�TLDR�generation.�CATTS� incorporates�ideas�from�scaffold�tasks�for�multitask� learning�(Swayamdipta�et�al.,�2018a;�Cohan�et�al.,� 2019)�and�control�codes�in�conditional�language� generation�(Keskar�et�al.,�2019)�to�address�the�prob- lem�of�data�scarcity�in�the�highly-specialized�sci- entiﬁc�domain.�In�particular,�CATTS�exploits�titles� as�an�auxiliary,� naturally-occurring�training�sig- nal�by�training�the�model�to�generate�both�titles� and�TLDRs�indicated�by�control�codes.� We�show� that�CATTS�applied�to�BART�(Lewis�et�al.,�2020),� a�state-of-the-art�summarization�model,�results�in� performance�improvement�in�both�automated�met- rics�and�human�evaluation.� Our�contributions�are�summarized�below:� 1.� We�introduce�TLDR� generation,�a�new�form� of�extreme�summarization,� for�scientiﬁc�papers.� With�extensive�analysis�of�properties�of�TLDRs,�we� provide�insight�into�the�types�of�information�and� amount�of�variability�in�human-written�TLDRs.� 2.� We� release� SCITLDR,� a� new� multi-target� dataset�of�5,411�TLDRs�over�3,229�scientiﬁc�papers.� SCITLDR�contains�both�author-written�and�expert- derived�TLDRs,�where�the�latter�are�collected�us- ing�a�novel�annotation�protocol�that�produces�high- quality�summaries�while�avoiding�the�burden�of� reading�the�full�paper.� 3.� We�establish�strong�baselines�on�SCITLDR�and� improve�them�with�CATTS,�a�simple�yet�effective� learning�strategy�for�generating�TLDRs�that�uses� titles�as�an�auxiliary�training�signal.� 4.� We�perform�extensive�analysis�and�human�eval- uation�of�system-generated� TLDRs,� focusing�on� informativeness�and�factual�correctness.� 2� Dataset�construction� Overview� We�introduce�SCITLDR,�a�new�multi- target�dataset�of�5,411�TLDRs�over�3,229�scientiﬁc� papers�in�the�computer�science�domain.3� The�train- ing�set�contains�1,992�papers,�each�with�a�single� gold�TLDR.�The�dev�and�test�sets�contain�619�and� 618�papers�each,�with�1,452�and�1,967�TLDRs,�re- spectively.�This�is�unlike�the�majority�of�existing� 3See�Appendix�Table�9�for�full�venue�breakdown.� Peer review The paper proposes variance regularizing ad- versarial learning (VRAL), a new method for training GANs. The motivation is to ensure that the gradient for the genera- tor does not vanish. [...] The discriminator itself is trained through two additional meta-discriminators Are the meta-dis- criminators really necessary? Have you tried matching moments or using other methods [...] Derived TLDR The paper proposes variance regularizing adversarial learning for training gans to ensure that the gradient for the generator does not vanish. 1 Figure�2:�Example�of�a�reviewer�comment�rewritten�as� a�TLDR�(best�viewed�in�color).�A�peer�review�comment� often�begins�with�a�summary�of�the�paper�which�anno- tators�use�to�compose�a�TLDR.� Annotators�are�trained� to�preserve�the�original�reviewer’s�wording�when�pos- sible�(indicated�by�colored�spans),�and�to�avoid�using� any�excess�details�or�criticism.� summarization�datasets�that�assume�only�one�gold� summary�for�a�given�document.� As�evidenced�by�earlier�work�in�summarization� evaluation�(Cohan�and�Goharian,�2016),�variabil- ity�in�human-written�summaries�(Zechner,�1996;� Harman�and�Over,�2004)�can�negatively�impact�the� reliability�of�automated�summarization�metrics�like� Rouge�(Lin,�2004).4� Considering�only�one�gold� TLDR�for�each�paper�as�a�basis�of�automated�eval- uation�might�result�in�inaccurate�system�quality� assessment�because�content�that�might�appear�in�a� TLDR�can�have�large�variability.�In�addition,�hav- …

Content truncated to 500 words

Labeled Text

Publication Details

Paper ID:: 29627
Year:: 2020
Publication:: TIPSTER
Publication Type:: conf
DOI:: nan

Links

Paper URL PDF Publication URL