WikiDragon: A Java Framework For Diachronic Content And Network Analysis Of MediaWikis

WikiDragon: A Java Framework For Diachronic Content And Network Analysis Of MediaWikis R¨udiger Gleim1, Alexander Mehler1, Sung Y. Song2 Text-Technology Lab, Goethe University1 Robert-Mayer-Straße 10, 60325 Frankfurt {gleim, mehler}@em.uni-frankfurt.de, gnosygnu@gmail.com2 Abstract We introduce WikiDragon, a Java Framework designed to give developers in computational linguistics an intuitive API to build, parse and analyze instances of MediaWikis such as Wikipedia, Wiktionary or WikiSource on their computers. It covers current versions of pages as well as the complete revision history, gives diachronic access to both page source code as well as accurately parsed HTML and supports the diachronic exploration of the page network. WikiDragon is self-enclosed and only requires an XML dump of the official Wikimedia Foundation website for import into an embedded database. No additional setup is required. We describe WikiDragon’s architecture and evaluate the framework based on the simple English Wikipedia with respect to the accuracy of link extraction, diachronic network analysis and the impact of using different Wikipedia frameworks to text analysis. Keywords: Wikipedia, Java Framework, Diachronic Text Analysis, Diachronic Network Analysis, Link Extraction 1. Introduction Wikis such as Wikipedia, Wiktionary, WikiSource or WikiNews constitute a popular form of collaborative writ- ing and thus a valuable resource for computational lin- guistics. This relates to research exploiting wiki data as a resource, e.g. for word sense disambiguation (Uslu et al., 2018; Dandala et al., 2013), explicit semantic analy- sis (Gabrilovich and Markovitch, 2007) or training data for named entity recognition (Nothman et al., 2008). Further- more, wikis are also considered as a research object on their own, for example with respect to measuring the impact of authors and edits (Priedhorsky et al., 2007), studies on dis- course structure (Mehler et al., 2018) as well as on edit categories (Daxenberger and Gurevych, 2012) or dynamics of conflicts in Wikipedias (Yasseri et al., 2012). The Wiki principle is also used to publish research results like digital editions of texts as Wikiditions (Mehler et al., 2016). Generally speaking, researchers need efficient as well as manageable means to access wiki data. However, one has to face the challenge that this data is not limited to arti- cles but also includes discussions, pages in other names- paces such as portals, rich metadata, revision histories, in- formation about (anonymous or registered) writers as well as about the link-based networking of pages. For small queries, using the Web API of wikis to collect data may be feasable. But in order to analyse large amounts of data (e.g. all articles of the main namespace) the approach to work online is inadequate because of the intense use of bandwith and server load. The alternative is to work offline based on local reconstructions of wikis. Notice that this scenario in- duces a big data problem: take the example of the German Wikipedia: as of 2016-02-03 its pages sum up to 23.84 GB of uncompressed data (current versions only). Its history pages sum-up to 2.92 TB of uncompressed data. Build- ing an offline instance of a wiki based on the MediaWiki software …

Content truncated to 500 words