Internet Argument Corpus 2.0: An SQL schema for Dialogic Social Media and the Corpora to go with it

Internet Argument Corpus 2.0: An SQL schema for Dialogic Social Media and the Corpora to go with it Rob Abbott, Brian Ecker, Pranav Anand, Marilyn A. Walker University of California, Santa Cruz abbott@soe.ucsc.edu, becker@ucsc.edu, panand@ucsc.edu, mawalker@ucsc.edu Abstract Large scale corpora have benefited many areas of research in natural language processing, but until recently, resources for dialogue have lagged behind. Now, with the emergence of large scale social media websites incorporating a threaded dialogue structure, content feedback, and self-annotation (such as stance labeling), there are valuable new corpora available to researchers. In previous work, we released the INTERNET ARGUMENT CORPUS, one of the first larger scale resources available for opinion sharing dialogue. We now release the INTERNET ARGUMENT CORPUS 2.0 (IAC 2.0) in the hope that others will find it as useful as we have. The IAC 2.0 provides more data than IAC 1.0 and organizes it using an extensible, repurposable SQL schema. The database structure in conjunction with the associated code facilitates querying from and combining multiple dialogically structured data sources. The IAC 2.0 schema provides support for forum posts, quotations, markup (bold, italic, etc), and various annotations, including Stanford CoreNLP annotations. We demonstrate the generalizablity of the schema by providing code to import the ConVote corpus. Keywords: dialogue, argument mining, sentiment, stance, data integration, online forums, debate 1. Introduction Large scale corpora have benefited many areas of re- search in natural language processing, but until recently, re- sources for dialogue have lagged behind. This is changing as more and more researchers work with social media sites structured in the form of dialogues, such as 4Forums, Cre- ate Debate and Reddit as well as sites such as Twitter that provide dialogic affordances such as synchronized commu- nity Tweet-Ups and the use of replies to other tweets (Abu- Jbara et al., 2012; Biran and Rambow, 2011; Somasundaran and Wiebe, 2009; Rosenthal and McKeown, 2015; Sridhar et al., 2015; Hassan et al., 2010; Cook et al., 2014; Cook et al., 2013; Bamman and Smith, 2015). In previous work, we released the INTERNET ARGUMENT CORPUS, one of the first larger scale resources available for opinion sharing dialogue (Walker et al., 2012b). Dataset Authors Discussions Posts Tokens 4forums 3.5K 11K 414K 57M ConvinceMe 5.5K 5.4K 65K 6.5M CreateDebate 709 61 3K 275K Table 1: The size of each dataset included. This paper describes the INTERNET ARGUMENT COR- PUS 2.0 (IAC 2.0).1 We have developed a larger scale dia- logic corpus by adding conversations from additional sites and structuring them into a novel data schema in SQL. See Table 1. The IAC 2.0 schema provides support for forum posts within distinct dialogues, quotations, markup (bold, italic, etc), and various human and machine developed an- notations, including Stanford CoreNLP annotations, such as POS tags, parses, and named entities. We provide Python code that facilitates querying and combining data from dif- ferent sources. We also demonstrate the generalizability of the schema with code to import the ConVote corpus (Thomas et al., 2006). 1IAC 2.0 is available at …

Content truncated to 500 words