Headerless, Quoteless, but not Hopeless? Using Pairwise Email Classification to Disentangle Email Threads

Proceedings of Recent Advances in Natural Language Processing, pages 327–335, Hissar, Bulgaria, 7-13 September 2013. Headerless, Quoteless, but not Hopeless? Using Pairwise Email Classification to Disentangle Email Threads Emily K. Jamison ‡ and Iryna Gurevych †‡ † Ubiquitous Knowledge Processing Lab (UKP-DIPF), German Institute for Educational Research and Educational Information Schloßstr. 29, 60486 Frankfurt, Germany ‡Ubiquitous Knowledge Processing Lab (UKP-TUDA), Department of Computer Science, Technische Universit¨at Darmstadt Hochschulstr. 10, 64289 Darmstadt, Germany http://www.ukp.tu-darmstadt.de Abstract Thread disentanglement is the task of sep- arating out conversations whose thread structure is implicit, distorted, or lost. In this paper, we perform email thread dis- entanglement through pairwise classifica- tion, using text similarity measures on non-quoted texts in emails. We show that i) content text similarity metrics out- perform style and structure text similar- ity metrics in both a class-balanced and class-imbalanced setting, and ii) although feature performance is dependent on the semantic similarity of the corpus, con- tent features are still effective even when controlling for semantic similarity. We make available the Enron Threads Corpus, a newly-extracted corpus of 70,178 multi- email threads with emails from the Enron Email Corpus. 1 Introduction Law enforcement agencies frequently obtain large amounts of electronic messages, such as emails, which they must search for evidence. However, individual messages may be useless without the conversational context they occur in. Most mod- ern emails contain useful metadata such as the MIME header In-Reply-To, which marks re- lations between emails in a thread and can be used to disentangle threads. However, there are easy methods of obfuscating email threads: opening an email account for a single purpose; using multi- ple email accounts for one person; sharing one email account among multiple persons; changing the Subject header; and removing quoted mate- rial from earlier in the thread. How can emails be organized by thread without metadata such as their MIME headers? We propose to use text similarity metrics to identify emails belonging to the same thread. In this paper, as a first step for temporal thread dis- entanglement, we perform pairwise classification experiments on texts in emails using no MIME headers or quoted previous emails. We have found that content-based text similarity metrics outper- form a Dice baseline, and that structural and style text similarity features do not; adding these lat- ter feature groups does not significantly improve total performance. We also found that content- based features continue to outperform the others in both a class-balanced and class-imbalanced set- ting, as well as with semantically controlled or non-controlled negative instances. In NLP, Elsner and Charniak (2010) described the task of thread disentanglement as “the cluster- ing task of dividing a transcript into a set of dis- tinct conversations,” in which extrinsic thread de- limitation is unavailable and the threads must be disentangled using only intrinsic information. In addition to emails with missing or incorrect MIME headers, entangled electronic conversations occur in environments such as interspersed Internet Re- lay Chat conversations, web 2.0 article response conversations that do not have a hierarchical dis- …

Content truncated to 500 words