Identification of Tasks, Datasets, Evaluation Metrics, and Numeric Scores for Scientific Leaderboards Construction

Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 5203–5213 Florence, Italy, July 28 - August 2, 2019. c⃝2019 Association for Computational Linguistics 5203 Identification of Tasks, Datasets, Evaluation Metrics, and Numeric Scores for Scientific Leaderboards Construction Yufang Hou, Charles Jochim, Martin Gleize, Francesca Bonin and Debasis Ganguly IBM Research – Ireland Dublin, Ireland {yhou|charlesj|mgleize|fbonin|debasga1}@ie.ibm.com Abstract While the fast-paced inception of novel tasks and new datasets helps foster active research in a community towards interesting directions, keeping track of the abundance of research ac- tivity in different areas on different datasets is likely to become increasingly difficult. The community could greatly benefit from an auto- matic system able to summarize scientific re- sults, e.g., in the form of a leaderboard. In this paper we build two datasets and develop a framework (TDMS-IE) aimed at automati- cally extracting task, dataset, metric and score from NLP papers, towards the automatic con- struction of leaderboards. Experiments show that our model outperforms several baselines by a large margin. Our model is a first step towards automatic leaderboard construction, e.g., in the NLP domain. 1 Introduction Recent years have witnessed a significant in- crease in the number of laboratory-based evalua- tion benchmarks in many of scientific disciplines, e.g., in the year 2018 alone, 140,616 papers were submitted to the pre-print repository arXiv1 and among them, 3,710 papers are under the Com- puter Science – Computation and Language cat- egory. This massive increase in evaluation bench- marks (e.g., in the form of shared tasks) is par- ticularly true for an empirical field such as NLP, which strongly encourages the research commu- nity to develop a set of publicly available bench- mark tasks, datasets and tools so as to reinforce reproducible experiments. Researchers have realized the importance of conducting meta-analysis of a number of compa- rable publications, i.e., the ones which use similar, if not identical, experimental settings, from shared tasks and proceedings, as shown by special issues 1https://arxiv.org/ dedicated to analysis of reproducibility in experi- ments (Ferro et al., 2018), or by detailed compar- ative analysis of experimental results reported on the same dataset in published papers (Armstrong et al., 2009). A useful output of this meta-analysis is often a summary of the results of a comparable set of experiments (in terms of the tasks they are ap- plied on, the datasets on which they are tested and the metrics used for evaluation) in a tabu- lar form, commonly referred to as a leaderboard. Such a meta-analysis summary in the form of a leaderboard is potentially useful to researchers for the purpose of (1) choosing the appropriate exist- ing literature for fair comparisons against a newly proposed method; and (2) selecting strong base- lines, which the new method should be compared against. Although recently there has been some effort to manually keep an account of progress on vari- ous research fields in the form of leaderboards, ei- ther by individual researchers2 or in a moderated crowd-sourced environment by organizations3, it is likely …

Content truncated to 500 words