Towards a Gold Standard Corpus for Variable Detection and Linking in Social Science Publications

Towards a Gold Standard Corpus for Variable Detection and Linking in Social Science Publications Andrea Zielinski and Peter Mutschke GESIS - Leibniz Institute for the Social Sciences {[andrea.zielinski,peter.mutschke]}@gesis.org Abstract In this paper, we describe our effort to create a new corpus for the evaluation of detecting and linking so-called survey variables in social science publications (e.g., ”Do you believe in Heaven?”). The task is to recognize survey variable mentions in a given text, disambiguate them, and link them to the corresponding variable within a knowledge base. Since there are generally hundreds of candidates to link to and due to the wide variety of forms they can take, this is a challenging task within NLP. The contribution of our work is the first gold standard corpus for the variable detection and linking task. We describe the annotation guidelines and the annotation process. The produced corpus is multilingual – German and English – and includes manually curated word and phrase alignments. Moreover, it includes text samples that could not be assigned to any variables, denoted as negative examples. Based on the new dataset, we conduct an evaluation of several state-of-the-art text classification and textual similarity methods. The annotated corpus is made available along with an open-source baseline system for variable mention identification and linking. Keywords: Text mining, semantic textual similarity, paraphrase detection, linking 1. Introduction There is a growing trend to integrate research data into the scientific publication process. Open Science encour- ages scientific practices in which all research data should be interlinked and contextualized to enhance reproducibil- ity and reusability of research results. Ideally, publications that report on a result of an empirical study should con- tain a direct link to the cited dataset and lead the reader directly to the research data that underlies the publication. However, in practice, this metadata is often missing. The potential of text and data mining technologies to automati- cally detect dataset citations has been addressed, e.g., in the International Workshop on Mining Scientific Publications1 and 4REAL Workshop2(Cohen et al., 2016; Branco, 2012; Fokkens et al., 2013). Interesting work in this direction has been carried out by Mariani et al. (2016) who seek to retrieve mentions of language resources (e.g., corpora, lexica listed in the LRE map) by analyzing the content of the proceedings of the Language Resources and Evaluation Conference (LREC), i.e., to discover the most relevant topics in this field. How- ever, specialized solutions for specific use cases are still re- quired within certain scientific areas. For instance, social sciences publications often discuss the results of survey studies. A survey generally consists of several hundreds of variables, each of them representing a single survey question (e.g., Do you believe in Heaven?). Social science papers, however, only focus on a particular selection of variables. In order to establish links between data and publications on a fine-grained level it is therefore necessary to link not only on study name level but also on the level of survey variables. The problem of automatically linking a data …

Content truncated to 500 words