The Inefficiency of Language Models in Scholarly Retrieval: An Experimental Walk-through

Findings of the Association for Computational Linguistics: ACL 2022, pages 3153 - 3173 May 22-27, 2022 c⃝2022 Association for Computational Linguistics The Inefficiency of Language Models in Scholarly Retrieval: An Experimental Walk-through Shruti Singh and Mayank Singh Department of Computer Science and Engineering Indian Institute of Technology Gandhinagar Gujarat, India {singh_shruti, singh.mayank}@iitgn.ac.in Abstract Language models are increasingly becoming popular in AI-powered scientific IR systems. This paper evaluates popular scientific lan- guage models in handling (i) short-query texts and (ii) textual neighbors. Our experiments showcase the inability to retrieve relevant doc- uments for a short-query text even under the most relaxed conditions. Additionally, we leverage textual neighbors, generated by small perturbations to the original text, to demon- strate that not all perturbations lead to close neighbors in the embedding space. Further, an exhaustive categorization yields several classes of orthographically and semantically related, partially related and completely unre- lated neighbors. Retrieval performance turns out to be more influenced by the surface form rather than the semantics of the text. 1 Introduction Representation learning methods have drastically evolved large scientific volume exploration strate- gies. The popular applications include sum- marization, construction of mentor-mentee net- work (Ke et al., 2021), recommendation (Osten- dorff et al., 2020; Cohan et al., 2020; Das et al., 2020; Hope et al., 2021), QA over scientific docu- ments (Su et al., 2020), and verification of scientific claims (Wadden et al., 2020). The growing com- munity’s interest has led to the development of sev- eral scientific document embedding models such as OAG-BERT (Liu et al., 2021), SPECTER (Cohan et al., 2020), SciBERT (Beltagy et al., 2019), and BioBERT (Lee et al., 2020) over the past five years. OAG-BERT has been deployed in the Aminer pro- duction system. Given similar possibilities of fu- ture deployments of scientific document embed- dings models in the existing scholarly systems, it is crucial to evaluate and identify limitations robustly. However, we do not find any existing work that critically analyzes the scientific language models to the best of our knowledge. To motivate the reader, we present a simple ex- periment. Queries ‘document vector’ and ‘docu- ment vectors’ fetch no common candidates among first page results on Google Scholar and Seman- tic Scholar (candidates in Appendix A). This il- lustrates the extremely brittle nature of such sys- tems to minor alterations in query text, leading to completely different search outcomes. To moti- vate further, we experiment with textual queries encoded by the popular SciBERT model. A per- turbed text ‘documen vector’ (relevant in AI) is closer to the Biomedical term ‘Virus vector’ in the embedding space. Similar observations were found for many other queries. As scholarly search and recommendation systems are complex systems and their detailed algorithms are not publicly available, we analyze the behavior of scientific language mod- els, which are (or will be) presumably an integral component of each of these systems. Motivated by the usage of perturbed inputs to stress test ML systems in interpretability analysis, we propose to use ‘textual neighbors’ to …

Content truncated to 500 words