Sparse Distillation: Speeding Up Text Classification by Using Bigger Student Models

Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 219 - 230 July 10-15, 2022 ©2022 Association for Computational Linguistics CERES: Pretraining of Graph-Conditioned Transformer for Semi-Structured Session Data Rui Feng†, Chen Luo‡, Qingyu Yin‡, Bing Yin‡, Tuo Zhao† and Chao Zhang† †Georgia Institute of Technology ‡Amazon Inc. {rfeng, tourzhao, chaozhang}@gatech.edu {cheluo,qingyy, alexbyin}@amazon.com Abstract User sessions empower many search and rec- ommendation tasks on a daily basis. Such session data are semi-structured, which en- code heterogeneous relations between queries and products, and each item is described by the unstructured text. Despite recent ad- vances in self-supervised learning for text or graphs, there lack of self-supervised learn- ing models that can effectively capture both intra-item semantics and inter-item interac- tions for semi-structured sessions. To fill this gap, we propose CERES, a graph-based transformer model for semi-structured ses- sion data. CERES learns representations that capture both inter- and intra-item seman- tics with (1) a graph-conditioned masked lan- guage pretraining task that jointly learns from item text and item-item relations; and (2) a graph-conditioned transformer architecture that propagates inter-item contexts to item- level representations. We pretrained CERES using ∼468 million Amazon sessions and find that CERES outperforms strong pretraining baselines by up to 9% in three session search and entity linking tasks. 1 Introduction User sessions are ubiquitous in online e-commerce stores. An e-commerce session contains customer interactions with the platform in a continuous pe- riod. Within one session, the customer can issue multiple queries and take various actions on the retrieved products for these queries, such as click- ing, adding to cart, and purchasing. Sessions are important in many e-commerce applications, e.g., product recommendation (Wu et al., 2019a), query recommendation (Cucerzan and White, 2007), and query understanding (Zhang et al., 2020). This paper considers sessions as semi-structured data, as illustrated in Figure 1. At the higher level, sessions are heterogeneous graphs that contain in- teractions between items. At the lower level, each Product 1 Title: Harrys Razor Type: Personal Care Product 2 Title: Harrys Potter Type: DVD Product 3 Title: Harrys Potter Type: Book Viewed Viewed Purchased Rewrite Rewrite Q1: "Harry" Q2: "Harry Potter" Q3: "Harry Potter Book" Figure 1: Illustration of a customer session. A session consists of two types of items: queries and products. The customer searched for 3 keywords sequentially and interacted with the products returned by the search en- gine. graph node has unstructured text descriptions: we can describe queries by search keywords and prod- ucts by titles, attributes, customer reviews, and other descriptors. Our goal is to simultaneously encode both the graph and text aspects of the ses- sion data to understand customer preferences and intents in a session context. Pretraining on semi-structured session data re- mains an open problem. First, existing works on learning from session data usually treat a session as a sequence or a graph (Xu et al., 2019; You et al., 2019; Qiu et al., 2020b). While they can model inter-item relations, …

Content truncated to 500 words