Using Local Knowledge Graph Construction to Scale Seq2Seq Models to Multi-Document Inputs

Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing, pages 4186–4196, Hong Kong, China, November 3–7, 2019. c⃝2019 Association for Computational Linguistics 4186 Using Local Knowledge Graph Construction to Scale Seq2Seq Models to Multi-Document Inputs Angela Fan FAIR / LORIA Claire Gardent CNRS / LORIA Chlo´e Braud CNRS / LORIA Antoine Bordes FAIR angelafan,abordes@fb.com claire.gardent,chloe.braud@loria.fr Abstract Query-based open-domain NLP tasks require information synthesis from long and diverse web results. Current approaches extrac- tively select portions of web text as input to Sequence-to-Sequence models using meth- ods such as TF-IDF ranking. We propose constructing a local graph structured knowl- edge base for each query, which compresses the web search information and reduces re- dundancy. We show that by linearizing the graph into a structured input sequence, models can encode the graph representations within a standard Sequence-to-Sequence setting. For two generative tasks with very long text in- put, long-form question answering and multi- document summarization, feeding graph rep- resentations as input can achieve better perfor- mance than using retrieved text portions. 1 Introduction Effective information synthesis is at the core of many Natural Language Processing applica- tions, such as open-domain question answering and multi-document summarization. In such tasks, a fundamental challenge is the ability to distill rel- evant knowledge from hundreds of thousands of tokens of noisy and redundant input such as web- pages. Current approaches predominantly conduct TF-IDF-based information extraction to identify key portions of the information, and then provide this as sequential input to a Sequence-to-Sequence (Seq2Seq) model. The sub-selected portions are limited to a few thousand words, as models often struggle to encode much longer sequences. In this work, we propose a mechanism to re- structure free text into local knowledge graphs that are then linearized into sequences, creating a canonical form in which information is presented to models. By constructing a graph intermedi- ary, redundant information can be merged or dis- carded, producing substantially compressed input Figure 1: Multi-Document Input to Linearized Graph Multi-document input resulting from web search queries are converted to a graph structured knowledge base using coref- erence resolution and information extraction, then linearized into a sequence for Seq2Seq models. Color indicates coref- erence resolution. Node weight is indicated by circle radius and edge weight by line thickness. — small enough to be fully encoded by Seq2Seq models. Such a method can be seen as merging previous work on symbolic knowledge bases for information extraction with newer approaches us- ing deep neural networks to encode knowledge. Our approach, shown in Figure 1, takes a query and its corresponding multi-document web search results and builds for each query a specific local knowledge graph. We present several modeling contributions to effectively encode the entire graph as a sequence and attend to the most relevant por- tions within this linearization. We demonstrate the effectiveness of this approach on two large- 4187 scale generative tasks with both long and noisy multi-document web input and paragraph length …

Content truncated to 500 words