MuLD: The Multitask Long Document Benchmark

Proceedings of the 13th Conference on Language Resources and Evaluation (LREC 2022), pages 3675–3685 Marseille, 20-25 June 2022 © European Language Resources Association (ELRA), licensed under CC-BY-NC-4.0 3675 MuLD: The Multitask Long Document Benchmark G Thomas Hudson, Noura Al Moubayed Durham University Department of Computer Science, Durham University, Durham, United Kingdom {g.t.hudson, noura.al-moubayed}@durham.ac.uk Abstract The impressive progress in NLP techniques has been driven by the development of multi-task benchmarks such as GLUE and SuperGLUE. While these benchmarks focus on tasks for one or two input sentences, there has been exciting work in designing efficient techniques for processing much longer inputs. In this paper, we present MuLD: a new long document benchmark consisting of only documents over 10,000 tokens. By modifying existing NLP tasks, we create a diverse benchmark which requires models to successfully model long-term dependencies in the text. We evaluate how existing models perform, and find that our benchmark is much more challenging than their ‘short document’ equivalents. Furthermore, by evaluating both regular and efficient transformers, we show that models with increased context length are better able to solve the tasks presented, suggesting that future improvements in these models are vital for solving similar long document problems. We release the data and code for baselines to encourage further research on efficient NLP models. Keywords: Long Documents, Benchmark, Multitask learning, NLP 1. Introduction Pretrained language models have been highly influen- tial in Natural Language Processing (NLP), leading to state-of-the-art results across a wide range of tasks. Based on the transformer architecture, these language models have shown capable at text classification, ques- tion answering, and translation among many other NLP problems. The rise of pretrained language models in NLP has been driven by influential benchmarks such GLUE (Wang et al., 2018), and SuperGLUE (Wang et al., 2019), which combine multiple existing datasets to provide a standardised evaluation of general-purpose Natural Language Understanding (NLU) approaches. Similar benchmarks have been created to evaluate mul- tilingual approaches (Yao et al., 2021), and other types of NLP task such as Natural Language Generation (NLG) (Liu et al., 2021). However, a key component of the success of the trans- former model - self-attention - is also a major limitation when it comes to processing longer sequences. Com- paring each token to all other tokens in the previous layer yields a O(n2) complexity, limiting the ability of standard transformers only a few hundred or thousand tokens on standard hardware. With many real world tasks involving the need to process documents in the range of tens of thousands of tokens, this is an impor- tant problem to solve. Recently, approaches such as Longformer (Beltagy et al., 2020), Reformer (Kitaev et al., 2020), and Lin- former (Wang et al., 2020) have explored techniques for improving the efficiency of transformers, allowing them to operate on much longer sequences. However, these have mainly been evaluated on artificial datasets, with limited evaluation on real-world data. There has been some work on creating long-text 0 10,000 20,000 30,000 40,000 50,000 60,000 Words per …

Content truncated to 500 words