Crawling and Preprocessing Mailing Lists At Scale for Dialog Analysis

Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 1151–1158 July 5 - 10, 2020. c⃝2020 Association for Computational Linguistics 1151 Crawling and Preprocessing Mailing Lists At Scale for Dialog Analysis Janek Bevendorff ∗ Khalid Al-Khatib∗ Martin Potthast‡ Benno Stein∗ ∗Bauhaus-Universität Weimar ‡Leipzig University <first>.<last>@uni-{weimar, leipzig}.de Abstract This paper introduces the Webis Gmane Email Corpus 2019, the largest publicly available and fully preprocessed email corpus to date. We crawled more than 153 million emails from 14,699 mailing lists and segmented them into semantically consistent components using a new neural segmentation model. With 96% ac- curacy on 15 classes of email segments, our model achieves state-of-the-art performance while being more efficient to train than previ- ous ones. All data, code, and trained models are made freely available alongside the paper.1 1 Introduction Email is perhaps the most reliable and ubiquitous means of digital communication. Notwithstanding the mainstream adoption of social media for private communication as of about 2010, email prevails un- rivaled for workplace communication and beyond. Compared to social media, however, emails have attracted much less research attention in the fields of computational linguistics, natural language pro- cessing, and information retrieval. Key reasons for the neglect can be found in the presumed diffi- culty of obtaining emails at scale, the lack of open technologies to parse them, and that, despite their importance, they are hardly considered en vogue. Although mailing lists as a rich and accessible source for emails have been tapped before, this has never been done at scale. Our contributions in this respect are (1) the Webis Gmane Email Crawl 2019, a crawl of more than 153 million emails from a wide range of mailing lists, (2) the Chipmunk email segmenter, a newly developed end-to-end neural model, and (3) the complete preprocessing of the crawled emails using our model to construct the largest corpus of “ready-to-use” emails to date. Our corpus encompasses more than 20 years worth of discussions on a diverse set of topics, including important political and societal issues. 1https://webis.de/publications.html?q=ACL+2020 We believe that providing the research commu- nity with access to clean and preprocessed commu- nication data from emails will foster open research in several areas, such as the analysis of dialogs and discourse, stylometry, language evolution, argu- ment mining, as well as information retrieval, and the synthesis of conversations and argumentation. 2 Related Work For research purposes, the three primary sources of email data are public mailing lists and newsgroups, volunteered or leaked private email datasets, and email databases at companies and service providers. The WestburyLab USENET corpus (Shaoul and Westbury, 2009, 2013) was crawled between 2005 and 2011. More widely employed has been the “20 newsgroups” corpus (Lang, 1995). The W3C corpus compiles the public W3C mailing lists (Wu, 2005), Jiang et al. (2013) examined 8 years of patch submissions to the Linux Kernel Mailing List, and Niedermayer et al. (2017) inspected the process of standardization across IETF bodies via its mail- ing lists. The CSpace corpus consists of 15,000 …

Content truncated to 500 words