Text Detoxification using Large Pre-trained Neural Models

Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 807–822 November 7–11, 2021. c⃝2021 Association for Computational Linguistics 807 Do Long-Range Language Models Actually Use Long-Range Context? Simeng Sun1 Kalpesh Krishna1 Andrew Mattarella-Micke2 Mohit Iyyer1 University of Massachusetts Amherst1 Intuit AI2 {simengsun,kalpesh,miyyer}@cs.umass.edu andrew_mattarella-micke@intuit.com Abstract Language models are generally trained on short, truncated input sequences, which limits their ability to use discourse-level information present in long-range context to improve their predictions. Recent efforts to improve the effi- ciency of self-attention have led to a prolifera- tion of long-range Transformer language mod- els, which can process much longer sequences than models of the past. However, the ways in which such models take advantage of the long- range context remain unclear. In this paper, we perform a fine-grained analysis of two long- range Transformer language models (includ- ing the Routing Transformer, which achieves state-of-the-art perplexity on the PG-19 long- sequence LM benchmark dataset) that accept input sequences of up to 8K tokens. Our re- sults reveal that providing long-range context (i.e., beyond the previous 2K tokens) to these models only improves their predictions on a small set of tokens (e.g., those that can be copied from the distant context) and does not help at all for sentence-level prediction tasks. Finally, we discover that PG-19 contains a va- riety of different document types and domains, and that long-range context helps most for lit- erary novels (as opposed to textbooks or mag- azines). 1 Introduction Understanding long documents requires model- ing various discourse-level phenomena, including anaphora (Hobbs, 1979; Grosz et al., 1995), argu- ment structure (Grimshaw, 1990), narrative scripts and trajectories (Schank and Abelson, 1977; Labov and Waletzky, 1997), and causal links between concepts (Mooney and DeJong, 1985). Unfortu- nately, most language models (LMs) are trained to predict the next word given only a small win- dow of local context, which prevents them from using long-range discourse structure to improve their predictions. Many research efforts over the years have attempted to address this issue: for ex- ample, Rosenfeld (1996) incorporated statistics from distant tokens to improve n-gram models, while Ji et al. (2015) added document-level context to neural LMs. More recently, the Transformer LM (Vaswani et al., 2017), which forms the backbone of state-of- the-art NLP systems (Devlin et al., 2019; Brown et al., 2020), has become the focus of numerous efforts to process longer input sequences. Trans- former LMs are constrained by the inefficiency of the self-attention mechanism, whose complexity scales quadratically with the input sequence length. As such, more efficient methods based on sparse attention (Correia et al., 2019) and cached mem- ory (Rae et al., 2020) have been proposed to in- crease the maximum sequence length, which has progressed from GPT-2’s 1024 tokens (Radford et al., 2019) to 4096 tokens (Zaheer et al., 2020) and finally 8192 tokens (Roy et al., 2021, Rout- ing Transformer). When evaluated on the PG-19 benchmark dataset (Rae et al., 2020), which con- tains long documents in the public domain, these “long-range” Transformer …

Content truncated to 500 words