Document Level Hierarchical Transformer

Document Level Hierarchical Transformer Najam Zaidi Monash University syed.zaidi1@monash.edu Gholamreza Haffari Monash University gholamreza.haffari@monash.edu Trevor Cohn University of Melbourne trevor.cohn@unimelb.edu.au Abstract Generating long and coherent text is an impor- tant and challenging task encompassing many application areas such as summarization, doc- ument level machine translation and story gen- eration. Despite the success in modeling intra- sentence coherence, existing long text gener- ation models (e.g., BART and GPT-3) still struggle to maintain a coherent event sequence throughout the generated text. We conjecture that this is because of the difficulty for the model to revise, replace or revoke any part that has been generated by the model. In this chapter, we present a novel semi- autoregressive document generation model ca- pable of revising and editing the generated text. Building on recent models by (Gu et al., 2019; Xu and Carpuat, 2020), we propose document generation as a hierarchical Markov decision process with a two level hierarchy, where the high and low level editing programs generate and refine the document. We train our model using imitation learning and introduce roll-in policy such that each policy learns on the out- put of applying the previous action. Experi- ments applying the proposed approach convey various insights on the problems of long text generation using our model. We suggest var- ious remedies such as using distilled dataset, designing better attention mechanisms and us- ing autoregressive models as a low level pro- gram. 1 Introduction Generating long and coherent text encompass vari- ous tasks such as summarization, story generation, document level machine translation and document level post editing. Each task is characterised by modelling long range dependencies to make the document coherent as well as modelling a high level plot to make the document thematically con- sistent (Fan et al., 2018). This is challenging as the models need to plan content, while producing local words consistent with the global context in a timely manner. Recent work on autoregressive generation mod- els, such as GPT-3 and BART (Lewis et al., 2019; Brown et al., 2020), have shown impressive per- formance in generating short fluent text with a maximum length ranging from 150 to 350 tokens (Bosselut et al., 2018; Shen et al., 2019; Zhao et al., 2020b). But applying the same model to generate longer passages of text (e.g., 1000 tokens) has re- sulted in syntactic and semantic errors throughout the document requiring extensive human curations (Tan et al., 2020). These massive language mod- els are usually pre-trained using large corpora of generic text, and then fine-tuned with small domain- specific data. Most of the time, the models are not publicly available to adapt to arbitrary desired do- mains. On the other hand, recent non-autoregressive ap- proaches allow generation to be done within a much smaller number of decoding iterations (Gu et al., 2017; Wang et al., 2019; Kasai et al., 2020). But due to its problems with modelling dependencies among the tokens, the approach still lags behind its autoregressive counterparts and has not yet been ap- plied …

Content truncated to 500 words