Modularized Transfomer-based Ranking Framework

Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing, pages 4180–4190, November 16–20, 2020. c⃝2020 Association for Computational Linguistics 4180 Modularized Transfomer-based Ranking Framework Luyu Gao Zhuyun Dai Jamie Callan Language Technologies Institute Carnegie Mellon University {luyug, zhuyund, callan}@cs.cmu.edu Abstract Recent innovations in Transformer-based ranking models have advanced the state-of- the-art in information retrieval. However, these Transformers are computationally expensive, and their opaque hidden states make it hard to understand the ranking process. In this work, we modularize the Transformer ranker into separate modules for text representation and interaction. We show how this design enables substantially faster ranking using offline pre-computed representations and light-weight online interactions. The modular design is also easier to interpret and sheds light on the ranking process in Transformer rankers.1 1 Introduction Neural rankers based on Transformer architectures (Vaswani et al., 2017) fine-tuned from BERT (Devlin et al., 2019) achieve current state-of-the- art (SOTA) ranking effectiveness (Nogueira and Cho, 2019; Craswell et al., 2019). The power of the Transformer comes from self-attention, the process by which all possible pairs of input tokens interact to understand their connections and contextualize their representations. Self-attention provides detailed, token-level information for matching, which is critical to the effectiveness of Transformer-based rankers (Wu et al., 2019). When used for ranking, a Transformer ranker takes in the concatenation of a query and docu- ment, applies a series of self-attention operations, and outputs from its last layer a relevance pre- diction (Nogueira and Cho, 2019). The entire ranker runs like a black box and hidden states have no explicit meanings. This represents a clear distinction from earlier neural ranking models that keep separate text representation and dis- tance (interaction) functions. Transformer rankers 1Open source code at https://github.com/ luyug/MORES are slow (Nogueira et al., 2019), and the black-box design makes it hard to interpret their behavior. We hypothesize that a Transformer- based ranker simultaneously performs text representation and query-document interaction as it processes the concatenated pair. Guided by this hypothesis, we decouple representation and interaction with a MOdualarized REranking System (MORES). MORES consists of three Transformer modules: the Document Representation Module, the Query Representation Module, and the Interaction Module. The two Representation Modules run independently of each other. The Document Representation Module uses self-attention to embed each document token conditioned on all document tokens. The Query Representation Module embeds each query token conditioned on all query tokens. The Interaction Module performs attention from query representations to document representations to generate match signals and aggregates them through self-attention over query tokens to make a relevance prediction. By disentangling the Transformer into modules for representation and interaction, MORES can take advantage of the indexing process: while the interaction must be done online, document repre- sentations can be computed offline. We further propose two strategies to pre-compute document representations that can be used by the Interaction Module for ranking. Our experiments on a large supervised rank- ing dataset demonstrate the effectiveness and ef- ficiency of MORES. It is as effective as a state- of-the-art …

Content truncated to 500 words