Simple Attention-Based Representation Learning for Ranking Short Social Media Posts

Proceedings of NAACL-HLT 2019, pages 2212–2217 Minneapolis, Minnesota, June 2 - June 7, 2019. c⃝2019 Association for Computational Linguistics 2212 Simple Attention-Based Representation Learning for Ranking Short Social Media Posts Peng Shi,1 Jinfeng Rao,2∗ and Jimmy Lin1 1 David R. Cheriton School of Computer Science, University of Waterloo 2 Facebook AI {peng.shi, jimmylin}@uwaterloo.ca, raojinfeng@fb.com Abstract This paper explores the problem of ranking short social media posts with respect to user queries using neural networks. Instead of start- ing with a complex architecture, we proceed from the bottom up and examine the effective- ness of a simple, word-level Siamese architec- ture augmented with attention-based mecha- nisms for capturing semantic “soft” matches between query and post tokens. Extensive experiments on datasets from the TREC Mi- croblog Tracks show that our simple models not only achieve better effectiveness than ex- isting approaches that are far more complex or exploit a more diverse set of relevance sig- nals, but are also much faster. Implemen- tations of our samCNN (Simple Attention- based Matching CNN) models are shared with the community to support future work.1 1 Introduction Despite a large body of work on neural rank- ing models for “traditional” ad hoc retrieval over web pages and newswire documents (Huang et al., 2013; Shen et al., 2014; Guo et al., 2016; Xiong et al., 2017; Mitra et al., 2017; Pang et al., 2017; Dai et al., 2018; McDonald et al., 2018), there has been surprisingly little work (Rao et al., 2017) on applying neural networks to searching short social media posts such as tweets on Twitter. Rao et al. (2019) identified short document length, informal- ity of language, and heterogeneous relevance sig- nals as main challenges in relevance modeling, and proposed the first neural model specifically designed to handle these characteristics. Eval- uation on a number of datasets from the TREC Microblog Tracks demonstrates state-of-the-art ef- fectiveness as well as the necessity of different model components to capture a multitude of rel- evance signals. ∗Work done at the University of Maryland, College Park. 1https://github.com/Impavidity/samCNN Query Embeddings Post Embeddings 𝑔" 𝑔# ℎ% 𝑣 General Sentence Encoder : QAtt or PAtt : Avg Aggregation : 𝑜 Figure 1: Our model architecture: a general sentence encoder is applied on query and post embeddings to generate gq and gp; an attention encoder is applied on post embeddings to generate variable-length query- aware features hi. These features are further aggre- gated to yield v, which feeds into the final prediction. In this paper, we also examine the problem of modeling relevance for ranking short social me- dia posts, but from a complementary perspective. As Weissenborn et al. (2017) notes, most sys- tems are built in a top-down process: authors pro- pose a complex architecture and then validate de- sign decisions with ablation experiments. How- ever, such experiments often lack comparisons to strong baselines, which raises the question as to whether model complexity is empirically justified. As an alternative, they advocate a bottom-up ap- proach where architectural complexity is gradu- ally increased. …

Content truncated to 500 words