Benchmarking a transformer-FREE model for ad-hoc retrieval

Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics, pages 3343–3353 April 19 - 23, 2021. ©2021 Association for Computational Linguistics 3343 Benchmarking a transformer-FREE model for ad-hoc retrieval Tiago Almeida IEETA Universidade de Aveiro Aveiro, Portugal tiagomeloalmeida@ua.pt S´ergio Matos DETI/IEETA Universidade de Aveiro Aveiro, Portugal aleixomatos@ua.pt Abstract Transformer-based “behemoths” have grown in popularity, as well as structurally, shat- tering multiple NLP benchmarks along the way. However, their real-world usability remains a question. In this work, we em- pirically assess the feasibility of applying transformer-based models in real-world ad-hoc retrieval applications by comparison to a “greener and more sustainable” alternative, comprising only 620 trainable parameters. We present an analysis of their efficacy and efficiency and show that considering limited computational resources, the lighter model running on the CPU achieves a 3 to 20 times speedup in training and 7 to 47 times in inference while maintaining a comparable retrieval performance. Code to reproduce the efficiency experiments is available on https: //github.com/bioinformatics-ua/ EACL2021-reproducibility/. 1 Introduction The Natural Language Processing (NLP) field has been revolutionised by the simple, yet extremely successful idea of transfer learning allied to the transformer neural architecture capability of ex- ploring long sequences, while being computa- tionally more efficient than the recurrent counter- parts. More precisely, Dai and Le (2015) and later Howard and Ruder (2018) introduced the transfer learning approach to the NLP field by pre-training a language model and then fine-tuning it to multiple NLP tasks. However, it was Radford (2018) who applied the same technique to a large transformer encoder, shattering the SOTA in 9 of 12 NLP tasks. Since then, the NLP field became dominated by works that use, explore or improve, in some way, the transformer-based architecture. Although their performance is undeniable, these are large models that comprise millions of parameters and require large computational resources to use and maintain, making them inoperable for the majority of smaller institutions. Furthermore, it also gives the impres- sion that the NLP field was reduced to throwing a lot of computation power (money) to continually achieve SOTA results in multiple benchmarks, leav- ing behind the careful process of designing a neural solution that follows the human intuition to solve some specific task. Moreover, given their running costs and the consequent impact on the environ- ment, their real-world virtues remain questionable, i.e., “is it really feasible to use these models in real-world applications, such as search engines or question answering systems to aid a broader range of people in their day-to-day tasks?”. As an exam- ple, biomedical experts need to routinely search an unprecedented amount of scientific literature to keep updated with their research, which could be facilitated by an intelligent system. In contradiction with this trend, this work exam- ines a lightweight interaction-based model, with only 620 trainable parameters, carefully designed by considering years of research in ad-hoc retrieval systems and interaction-based architectures. This system was evaluated in two ad-hoc retrieval com- petitions where it was able to compete with …

Content truncated to 500 words