Extracting Structured Scholarly Information from the Machine Translation Literature

Extracting Structured Scholarly Information from the Machine Translation Literature Eunsol Choi†, Matic Horvat†, Jonathan May∗, Kevin Knight∗, Daniel Marcu∗ University of Washington, University of Cambridge, Information Sciences Institute∗ eunsol@cs.washington.edu, matic.horvat@cl.cam.ac.uk, {may,knight,marcu}@isi.edu Abstract Understanding the experimental results of a scientific paper is crucial to understanding its contribution and to comparing it with related work. We introduce a structured, queryable representation for experimental results and a baseline system that automatically populates this representation. The representation can answer compositional questions such as: “Which are the best published results reported on the NIST 09 Chinese to English dataset?” and “What are the most important methods for speeding up phrase-based decoding?” Answering such questions usually involves lengthy literature surveys. Current machine reading for academic papers does not usually consider the actual experiments, but mostly focuses on understanding abstracts. We describe annotation work to create an initial ⟨scientific paper; experimental results representation⟩ corpus. The corpus is composed of 67 papers which were manually annotated with a structured representation of experimental results by domain experts. Additionally, we present a baseline algorithm that characterizes the difficulty of the inference task. Keywords: Information Extraction, Scientific Literature, Structured Prediction 1. Introduction Current technologies enable one to access large scientific literature repositories via a variety of means, which range from simple keyword searches for content and authors to sophisticated inferences that exploit citation links (Dunne et al., 2010; Sch¨afer et al., 2011), techniques that auto- matically identify sections and section labels (Teufel and Kan, 2011), and unsupervised methods to infer information structures (Kiela et al., 2015). Unfortunately, these access methods fall short of supporting many queries that could sig- nificantly improve the day-to-day activities of a researcher. Imagine, for example, a young researcher who wants to be- gin working on Machine Translation (MT) or a seasoned researcher who wants to keep track of recent developments in the field. Ideally, they would like to quickly get answers to questions like: • Which are the best published results reported on the NIST-09 Chinese dataset? • What are the papers that show on training sets larger than 100M words that morphology-inspired models lead to improvements in translation quality that are statistically significant? • What are the most important methods for speeding up phrase-based decoding? • Are there papers showing that a neural translation model is better than a non-neural model? To our knowledge, answering such queries is beyond the state of the art. Current methods cannot yet infer the main elements of experiments reported in papers; as a matter of fact, no consensus exists on what these elements should be and what the relations between them are. In this paper, we take a few steps towards addressing these shortcomings. By focusing on MT as our exemplary sub- field of study, we propose a representation that explicitly models the hidden structure of typical experiments: data sources used for training and testing, evaluation metrics, † Work done during internship at ISI. Both authors contributed equally. languages, baseline algorithms, methods and algorithms that experiments are meant to …

Content truncated to 500 words