Cross-Domain Modeling of Sentence-Level Evidence for Document Retrieval

Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing, pages 3490–3496, Hong Kong, China, November 3–7, 2019. c⃝2019 Association for Computational Linguistics 3490 Cross-Domain Modeling of Sentence-Level Evidence for Document Retrieval Zeynep Akkalyoncu Yilmaz, Wei Yang, Haotian Zhang, and Jimmy Lin David R. Cheriton School of Computer Science University of Waterloo Abstract This paper applies BERT to ad hoc document retrieval on news articles, which requires ad- dressing two challenges: relevance judgments in existing test collections are typically pro- vided only at the document level, and docu- ments often exceed the length that BERT was designed to handle. Our solution is to ag- gregate sentence-level evidence to rank docu- ments. Furthermore, we are able to leverage passage-level relevance judgments fortuitously available in other domains to fine-tune BERT models that are able to capture cross-domain notions of relevance, and can be directly used for ranking news articles. Our simple neural ranking models achieve state-of-the-art effec- tiveness on three standard test collections. 1 Introduction The dominant approach to ad hoc document re- trieval using neural networks today is to deploy the neural model as a reranker over an initial list of candidate documents retrieved using a standard bag-of-words term-matching technique. Despite the plethora of neural models that have been pro- posed for document ranking (Mitra and Craswell, 2019), there has recently been some skepticism about whether they have truly advanced the state of the art (Lin, 2018), at least in the absence of large amounts of behavioral log data only avail- able to search engine companies. In a meta-analysis of over 100 papers that re- port results on the dataset from the Robust Track at TREC 2004 (Robust04), Yang et al. (2019a) found that most neural approaches do not com- pare against competitive baselines. To provide two recent examples, McDonald et al. (2018) report a best AP score of 0.272 and Li et al. (2018) 0.290, compared to a simple bag-of-words query expan- sion baseline that achieves 0.299 (Lin, 2018). Fur- ther experiments by Yang et al. (2019a) achieve 0.315 under more rigorous experimental condi- tions with a neural ranking model, but this is still pretty far from the best-known score of 0.3686 on this dataset (Cormack et al., 2009). Although Sculley et al. (2018) remind us that the goal of science is not wins, but knowledge, the latter requires first establishing strong base- lines that accurately quantify proposed contribu- tions. Comparisons to weak baselines that inflate the merits of an approach are not new problems in information retrieval (Armstrong et al., 2009), and researchers have in fact observed similar issues in the recommender systems literature as well (Ren- dle et al., 2019; Dacrema et al., 2019). Having placed evaluation on more solid footing with respect to well-tuned baselines by building on previous work, this paper examines how we might make neural approaches “work” for document re- trieval. One promising recent innovation is mod- els that exploit massive pre-training …

Content truncated to 500 words