A Crowdsourcing Approach for Annotating Causal Relation Instances in Wikipedia

31st Pacific Asia Conference on Language, Information and Computation (PACLIC 31), pages 336–345 Cebu City, Philippines, November 16-18, 2017 Copyright c⃝2017 Kazuaki Hanawa, Akira Sasaki, Naoaki Okazaki and Kentaro Inui A Crowdsourcing Approach for Annotating Causal Relation Instances in Wikipedia Kazuaki Hanawa, Akira Sasaki, Naoaki Okazaki, Kentaro Inui Tohoku University {hanawa, aki-s, okazaki, inui}@ecei.tohoku.ac.jp Abstract This paper presents a crowdsourcing approach for annotating causal relation instances to Wikipedia. Because an annotation task can- not be decomposed into multiple-choice prob- lems, we integrate a crowdsourcing service and brat, a popular on-line annotation tool, to provide an easy-to-use interface and quality control for annotation work. We design sim- ple micro-tasks that involve annotating tex- tual spans with causal relations. We issued the micro-tasks to crowd workers, and col- lected 95,008 annotations of causal relation instances among 8,745 summary sentences in 1,494 Wikipedia articles. The annotated cor- pus not only provides supervision data for automatic recognition of causal relation in- stances, but also reveals valuable facts for im- proving the annotation process of this task. 1 Introduction Commonsense knowledge such as entities and events, and their causal relationships, are indis- pensable in various natural language processing (NLP) applications, including question answer- ing (Oh et al., 2013; Oh et al., 2016; Sharp et al., 2016), hypothesis generation (Radinsky et al., 2012; Hashimoto et al., 2015), stance detection (Sasaki et al., 2016), and literature curation for systems biol- ogy (Pyysalo et al., 2015; Rinaldi et al., 2016). In many previous researches, corpora for acquir- ing causal relations were built by annotating two text spans (e.g., entities) and their relations in the text (Doddington et al., 2004; Hendrickx et al., 2010; Pyysalo et al., 2015; Rinaldi et al., 2016; Dunietz et al., 2017; Rehbein and Ruppenhofer, 2017). However, this approach is extremely work intensive. It involves choosing a target domain, de- signing an ontology (semantic classes) of entities, building a corpus for named entity recognition, de- signing an annotation guideline for relations, and annotating the relations between entities. Building such a corpus also requires the annotation efforts of experts. For these reasons, this approach is al- most non-scalable to various domains or genres of text although the knowledge of the causal relations is highly target-specific. This paper presents an approach for harness- ing causal relation instances to Wikipedia articles via crowdsourcing. Wikipedia is the central in- frastructure for knowledge curation, as exemplified by Freebase (Bollacker et al., 2008) and Wikifica- tion (Mihalcea and Csomai, 2007). Therefore, we base Wikipedia articles for building a corpus with causal relation instances. This work represents a first step toward organizing the causal knowledge in Wikipedia articles covering various topics. Recently, researchers have recognized the value of crowdsourcing services in constructing wide- ranging language resources at low cost (Brew et al., 2010; Finin et al., 2010; Gormley et al., 2010; Jha et al., 2010; Fort et al., 2011; Kawahara et al., 2014; Lawson et al., 2010; Hovy et al., 2014; Takase et al., 2016). Unfortunately, causal relations cannot be directly annotated …

Content truncated to 500 words