Assist Non-native Viewers: Multimodal Cross-Lingual Summarization for How2 Videos

Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 730 - 744 December 7-11, 2022 ©2022 Association for Computational Linguistics ReSel: N-ary Relation Extraction from Scientific Text and Tables by Learning to Retrieve and Select Yuchen Zhuang1, Yinghao Li1, Jerry Junyang Cheung1, Yue Yu1, Yingjun Mou1, Xiang Chen2, Le Song3,4, Chao Zhang1 1Georgia Institute of Technology, Atlanta, USA 2Adobe Research, San Jose, USA 3MBZUAI, Abu Dhabi, UAE 4BioMap, Beijing, China 1{yczhuang,yinghaoli,jzhang3027,yueyu,ymou32, chaozhang}@gatech.edu 2xiangche@adobe.com 3le.song@mbzuai.ac.ae Abstract We study the problem of extracting N-ary re- lation tuples from scientific articles. This task is challenging because the target knowledge tuples can reside in multiple parts and modal- ities of the document. Our proposed method RESEL decomposes this task into a two-stage procedure that first retrieves the most relevant paragraph/table and then selects the target en- tity from the retrieved component. For the high- level retrieval stage, RESEL designs a simple and effective feature set, which captures multi- level lexical and semantic similarities between the query and components. For the low-level selection stage, RESEL designs a cross-modal entity correlation graph along with a multi-view architecture, which models both semantic and document-structural relations between entities. Our experiments on three scientific information extraction datasets show that RESEL outper- forms state-of-the-art baselines significantly. 1 1 Introduction Scientific information extraction (SciIE) (Augen- stein et al., 2017; Luan et al., 2018; Jiang et al., 2019), the task of extracting scientific concepts along with their relations from scientific literature corpora, is important for researchers to keep abreast of latest scientific advances. A key subtask of SciIE is the N-ary relation extraction problem (Jia et al., 2019; Jain et al., 2020), which aims to extract the relations of different entities as N-ary knowledge tuples. This problem is challenging because the entities of the knowledge tuples often reside in multiple sections (e.g., abstracts, experiments) and modalities (e.g., paragraphs, tables, figures) of the document. Effective scientific N-ary relation ex- traction requires not only understanding the seman- tics of different modalities, but also performing document-level inference based on interleaving sig- 1Our code is available on https://github.com/ night-chen/ReSel. Query 1: <Dependency Parsing, English PTB, Arc- hybrid (dyn, α=0.75), LAS> Answer 1: 91.42 Support 1: Query 2: <Dependency Parsing, English PTB, Arc- hybrid (dyn, α=0.75), UAS> Answer 2: 93.56 Support 2: Scientific Literature Query Examples Figure 1: Illustration of the multi-modal scientific N- ary relation extraction problem on the SciREX dataset. nals such as co-occurrences, co-references, and structural relations, as shown in Figure 1. Document-level N-ary relation extraction has been studied in literature (Jia et al., 2019; Jain et al., 2020; Viswanathan et al., 2021; Liu et al., 2021). Some works (Zeng et al., 2020; Tu et al., 2019) use graph-based approaches to model long-distance relations in the document with the focus on text only. However, for scientific articles, an equally if not more important data structure is the table, as scientific results are often reported in tables and then referred to and discussed in text. There are also works that pre-train …

Content truncated to 500 words