DiPair: Fast and Accurate Distillation for Trillion-Scale Text Matching and Pair Modeling

Findings of the Association for Computational Linguistics: EMNLP 2020, pages 2925–2937 November 16 - 20, 2020. c⃝2020 Association for Computational Linguistics 2925 DiPair: Fast and Accurate Distillation for Trillion-Scale Text Matching and Pair Modeling Jiecao Chen∗, Liu Yang, Karthik Raman, Michael Bendersky Jung-Jung Yeh, Yun Zhou, Marc Najork, Danyang Cai, Ehsan Emadzadeh Google Research Abstract Pre-trained models like BERT (Devlin et al., 2018) have dominated NLP / IR applications such as single sentence classification, text pair classification, and question answering. How- ever, deploying these models in real systems is highly non-trivial due to their exorbitant computational costs. A common remedy to this is knowledge distillation (Hinton et al., 2015), leading to faster inference. However – as we show here – existing works are not opti- mized for dealing with pairs (or tuples) of texts. Consequently, they are either not scalable or demonstrate subpar performance. In this work, we propose DiPair — a novel framework for distilling fast and accurate models on text pair tasks. Coupled with an end-to-end training strategy, DiPair is both highly scalable and of- fers improved quality-speed tradeoffs. Empir- ical studies conducted on both academic and real-world e-commerce benchmarks demon- strate the efficacy of the proposed approach with speedups of over 350x and minimal qual- ity drop relative to the cross-attention teacher BERT model. 1 Introduction Modeling the relationship between textual objects is critical to numerous NLP and information re- trieval (IR) applications (Li and Xu, 2014). This subsumes a number of different problems such as textual entailment, semantic text matching, para- phrase identification, plagiarism detection, and rel- evance modeling. For example, modeling the rela- tionship between queries and documents / ad key- words is central to search engines / digital advertise- ment systems (Li and Xu, 2014; Guo et al., 2019). Recently neural network-based models have demonstrated large gains in this space (Hu et al., 2014; Pang et al., 2016). In particular, the Trans- former / BERT family of models (Devlin et al., ∗Correspondence to chenjiecao@google.com 2018; Lan et al., 2019; Liu et al., 2019; Clark et al., 2020) have set a new bar for these seman- tic text matching problems. However, the compu- tational costs of these models have proven to be prohibitively expensive, thus limiting their use in real-world applications (Frankle and Carbin, 2019). For example, on the e-commerce relevance-scoring task (P2T-REL dataset) discussed in Sec. 4.1, scor- ing the (trillion+) text pairs would take years. One popular remedy is to distill these expen- sive teacher models (Hinton et al., 2015) into lightweight student models. Training these students using examples labeled by the teacher has been shown to maintain quality while enabling faster inference. The key to the effectiveness of distilla- tion techniques is a good trade-off between student quality and inference speed. However, as we show here, existing knowledge distillation techniques (Sanh et al., 2019; Jiao et al., 2019; Turc et al., 2019; Tang et al., 2019) fall short on the quality-speed trade-off when dealing with pairs of texts. On one hand, approaches …

Content truncated to 500 words