LOT: A Story-Centric Benchmark for Evaluating Chinese Long Text Understanding and Generation

LOT: A Story-Centric Benchmark for Evaluating Chinese Long Text Understanding and Generation Jian Guan1, Zhuoer Feng1, Yamei Chen1, Ruilin He2, Xiaoxi Mao3, Changjie Fan3, Minlie Huang1∗ 1The CoAI group, DCST, China; 2Huawei Technologies Co., Ltd., China; 3Netease Fuxi AI Lab., China {j-guan19,fze17}@mails.tsinghua.edu.cn, chenziym4132013@163.com, heruilin@huawei.com, {maoxiaoxi,fanchangjie}@corp.netease.com, aihuang@tsinghua.edu.cn Abstract Standard multi-task benchmarks are essen- tial for developing pretraining models that can generalize to various downstream tasks. Existing benchmarks for natural language processing (NLP) usually focus only on under- standing or generating short texts. However, long text modeling requires many distinct abil- ities in contrast to short texts, such as the modeling of long-range discourse and com- monsense relations, and the coherence and controllability of generation. The lack of stan- dardized benchmarks makes it difficult to assess these abilities of a model and fairly compare different models, especially Chinese models. Therefore, we propose a story-centric benchmark named LOT for evaluating Chi- nese long text modeling, which aggregates two understanding tasks and two generation tasks. We construct new datasets for these tasks based on human-written Chinese sto- ries with hundreds of words. Furthermore, we release an encoder-decoder-based Chinese long text pretraining model named LongLM with up to 1 billion parameters. We pre- train LongLM on 120G Chinese novels with two generative tasks including text infilling and conditional continuation. Extensive ex- periments show that LongLM outperforms similar-sized pretraining models substantially on both the understanding and generation tasks in LOT. 1 Introduction Pretrained language models have achieved sig- nificant advances in various natural language understanding (NLU) and generation (NLG) tasks ∗ Corresponding author. (Devlin et al., 2019; Radford et al., 2019). Standard benchmarks such as GLUE (Wang et al., 2019) further boost the improvement and fast iteration of pretrained models. Popular benchmarks usually aggregate multiple tasks to spur the progress of generalizable models. But these benchmarks fo- cus mainly on understanding or generating short texts. For example, the GLUE tasks take at most two sentences as input. And most tasks in NLG benchmarks such as GLGE (Liu et al., 2020) and GEM (Gehrmann et al., 2021) require generating only several words (e.g., dialogue generation). Al- though there have been many models pretrained on long texts such as GPT3 (Brown et al., 2020) and CPM (Zhang et al., 2020), the lack of bench- mark datasets makes it difficult to fully assess and compare their abilities of long text modeling. In this paper, we present LOT, a benchmark for evaluating Chinese LOng Text understanding and generation. As shown in Table 1, modeling long texts requires many distinct abilities compared to short texts, including (1) commonsense reasoning regarding characters’ reaction and intention, and knowledge about physical objects (e.g., ‘‘river’’) and abstract concepts (e.g., ‘‘irony’’); (2) model- ing discourse-level features such as inter-sentence relations (e.g., causality) and global discourse structures (e.g., the order of events); and (3) the generation coherence and controllability, which require both maintaining a coherent plot and adhering to controllable attributes (e.g., topics). Accordingly, LOT contains two understanding tasks and two generation tasks regarding the above abilities. …

Content truncated to 500 words