Long Document Ranking with Query-Directed Sparse Transformer

Findings of the Association for Computational Linguistics: EMNLP 2020, pages 4594–4605 November 16 - 20, 2020. c⃝2020 Association for Computational Linguistics 4594 Long Document Ranking with Query-Directed Sparse Transformer Jyun-Yu Jiang†, Chenyan Xiong‡, Chia-Jung Lee§ and Wei Wang† †Department of Computer Science, University of California, Los Angeles, USA ‡Microsoft Research AI, Redmond, USA §Amazon, Seattle, USA {jyunyu,weiwang}@cs.ucla.edu, chenyan.xiong@microsoft.com, cjlee@amazon.com Abstract The computing cost of transformer self- attention often necessitates breaking long doc- uments to fit in pretrained models in docu- ment ranking tasks. In this paper, we de- sign Query-Directed Sparse attention that in- duces IR-axiomatic structures in transformer self-attention. Our model, QDS-Transformer, enforces the principle properties desired in ranking: local contextualization, hierarchi- cal representation, and query-oriented prox- imity matching, while it also enjoys effi- ciency from sparsity. Experiments on one fully supervised and three few-shot TREC document ranking benchmarks demonstrate the consistent and robust advantage of QDS- Transformer over previous approaches, as they either retrofit long documents into BERT or use sparse attention without emphasizing IR principles. We further quantify the computing complexity and demonstrates that our sparse attention with TVM implementation is twice more efficient that the fully-connected self- attention. All source codes, trained model, and predictions of this work are available at https://github.com/hallogameboy/ QDS-Transformer. 1 Introduction Pre-trained Transformers such as BERT (Devlin et al., 2019) effectively transfer language under- standing to better relevance estimation in many search ranking tasks (Nogueira and Cho, 2019; Nogueira et al., 2019; Yang et al., 2019). Nev- ertheless, the effectiveness comes at the quadratic cost O(n2) in computing complexity corresponds to the text length n, prohibiting its direct appli- cation to long documents. Prior work adopts quick workarounds such as document truncation or splitting-and-pooling to retrofit the document ranking task to pretrained transformers. Whilst there have been successes with careful architecture Figure 1: An example illustration of the attention mech- anism used in Query-Directed Sparse Transformer. design, those bandit-solutions inevitably introduce information loss and create complicated system pipelines. Intuitively, effective document ranking does not require fully connected self-attention between all query and document terms. The relevance match- ing between queries and documents often takes place at text segments as opposed to individual tokens (Callan, 1994; Jiang et al., 2019), suggest- ing that a document term may not need informa- tion thousands of words away (Metzler and Croft, 2005; Child et al., 2019), and that not all document terms are useful to calculate the relevance to the query (Xiong et al., 2017). The fully connected at- tention matrix includes many unlikely connections that create efficiency debt in computing, inference time, parameter size, and training convergence. This paper presents Query-Directed Sparse Transformer (QDS-Transformer) for long docu- ment ranking. In contrast to retrofitted solutions, QDS-Transformer fundamentally considers the de- sirable properties for assessing relevance by focus- ing on attention paths that matter. Using sparse local attention (Child et al., 2019), our model re- moves unnecessary connections between distant 4595 document tokens. Using global attention upon sentence boundaries, our model further incorpo- rates the hierarchical structures …

Content truncated to 500 words