LoRaLay: A Multilingual and Multimodal Dataset for Long Range and Layout-Aware Summarization

Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics, pages 636–651 May 2-6, 2023 ©2023 Association for Computational Linguistics LoRaLay: A Multilingual and Multimodal Dataset for Long Range and Layout-Aware Summarization Laura Nguyen1,3 Thomas Scialom2∗ Benjamin Piwowarski3 Jacopo Staiano4∗ 1reciTAL, Paris, France 2Meta AI, Paris, France 3Sorbonne Université, CNRS, ISIR, F-75005 Paris, France 4University of Trento, Italy laura@recital.ai benjamin.piwowarski@cnrs.frtscialom@fb.com jacopo.staiano@unitn.it Abstract Text Summarization is a popular task and an active area of research for the Natural Lan- guage Processing community. It requires ac- counting for long input texts, a characteristic which poses computational challenges for neu- ral models. Moreover, real-world documents come in a variety of complex, visually-rich, layouts. This information is of great relevance, whether to highlight salient content or to en- code long-range interactions between textual passages. Yet, all publicly available summa- rization datasets only provide plain text con- tent. To facilitate research on how to exploit visual/layout information to better capture long- range dependencies in summarization models, we present LoRaLay, a collection of datasets for long-range summarization with accompa- nying visual/layout information. We extend existing and popular English datasets (arXiv and PubMed) with visual/layout information and propose four novel datasets – consistently built from scholar resources – covering French, Spanish, Portuguese, and Korean languages. Further, we propose new baselines merging layout-aware and long-range models – two or- thogonal approaches – and obtain state-of-the- art results, showing the importance of combin- ing both lines of research. 1 Introduction Deep learning techniques have enabled remarkable progress in Natural Language Processing (NLP) in recent years (Devlin et al., 2018; Raffel et al., 2019; Brown et al., 2020). However, the majority of models, benchmarks, and tasks have been de- signed for unimodal approaches, i.e. focusing ex- clusively on a single source of information, namely plain text. While it can be argued that for specific NLP tasks, such as textual entailment or machine translation, plain text is all that is needed, there exist several tasks for which disregarding the vi- sual appearance of text is clearly sub-optimal: in *Work partially done while at reciTAL. a real-world context (business documentation, sci- entific articles, etc.), text does not naturally come as a sequence of characters, but is rather displayed in a bi-dimensional space containing rich visual information. The layout of e.g. this very paper provides valuable semantics to the reader: in which section are we right now? At the blink of an eye, this information is readily accessible via the salient section title (formatted differently and placed to highlight its role) preceding these words. Just to emphasize this point, imagine having to scroll this content in plain text to access such information. In the last couple of years, the research commu- nity has shown a growing interest in addressing these limitations. Several approaches have been proposed to deal with visually-rich documents and integrate layout information into language mod- els, with direct applications to Document Under- standing tasks. Joint multi-modal pretraining (Xu et al., 2021; Powalski et al., 2021; …

Content truncated to 500 words