AxCell: Automatic Extraction of Results from Machine Learning Papers

Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing, pages 8580–8594, November 16–20, 2020. c⃝2020 Association for Computational Linguistics 8580 AXCELL: Automatic Extraction of Results from Machine Learning Papers Marcin Kardas1 Piotr Czapla2 Pontus Stenetorp3 Sebastian Ruder4 Sebastian Riedel1,3 Ross Taylor1 Robert Stojnic1 1Facebook AI Research, {mkardas,sriedel,rjt,rstojnic}@fb.com 2n-waves, piotr.czapla@n-waves.com 3University College London, p.stenetorp@cs.ucl.ac.uk 4DeepMind, ruder@google.com Abstract Tracking progress in machine learning has be- come increasingly difficult with the recent ex- plosion in the number of papers. In this pa- per, we present AXCELL, an automatic ma- chine learning pipeline for extracting results from papers. AXCELL uses several novel com- ponents, including a table segmentation sub- task, to learn relevant structural knowledge that aids extraction. When compared with ex- isting methods, our approach significantly im- proves the state of the art for results extraction. We also release a structured, annotated dataset for training models for results extraction, and a dataset for evaluating the performance of mod- els on this task. Lastly, we show the viabil- ity of our approach enables it to be used for semi-automated results extraction in produc- tion, suggesting our improvements make this task practically viable for the first time. Code is available on GitHub.1 1 Introduction Machine learning studies how machines learn with respect to a task, a performance metric, and a dataset (Mitchell, 2006). The (task, dataset, metric name, metric value) tuple can therefore be seen as representing a single result of a machine learning paper. To make progress as a field we need to make comparisons between results achieved with differ- ent methodologies. In light of the explosion in the number of machine learning publications in recent years, such comparisons have become more diffi- cult.2 This poses serious challenges to peer review, among others. For instance, across ten language modelling papers submitted to ICLR 2018, the per- plexity score of the best baseline differed by more 1https://github.com/paperswithcode/ axcell 2In 2019, over 33,000 machine learning papers were pub- lished on the arXiv.org open-access e-print archive, with a year-on-year growth of around 50% since 2015. than 50 points (Ruder, 2018). One way to deal with the deluge of papers is to develop automatic approaches for extracting re- sults from papers and aggregating them into leader- boards. Authors typically publish their results in a tabular format in the paper, including a selection of comparisons between their approach and past papers. Automatic extraction of result tuples from tables—and optionally metadata such as model names—enables a full comparison between pub- lished methods. Online leaderboards for comparison have be- come increasingly common in the research com- munity. But these are only available for a few tasks and do not aid the comparison of models across tasks. To fill the gap, result aggregation tools such as Papers With Code3 and NLP-Progress4 utilise crowdsourced community contributions to populate paper leaderboards. However, human annotation of results can be laborious and error-prone, lead- ing to omission or misreporting of paper results. Automating at least some parts of the process can speed-up the …

Content truncated to 500 words