Combining Probability-Based Rankers for Action-Item Detection

Proceedings of NAACL HLT 2007, pages 324–331, Rochester, NY, April 2007. c⃝2007 Association for Computational Linguistics Combining Probability-Based Rankers for Action-Item Detection Paul N. Bennett Microsoft Research∗ One Microsoft Way Redmond, WA 98052 paul.n.bennett@microsoft.com Jaime G. Carbonell Language Technologies Institute, Carnegie Mellon 5000 Forbes Ave Pittsburgh, PA 15213 jgc@cs.cmu.edu Abstract This paper studies methods that automat- ically detect action-items in e-mail, an important category for assisting users in identifying new tasks, tracking ongoing ones, and searching for completed ones. Since action-items consist of a short span of text, classifiers that detect action-items can be built from a document-level or a sentence-level view. Rather than com- mit to either view, we adapt a context- sensitive metaclassification framework to this problem to combine the rankings pro- duced by different algorithms as well as different views. While this framework is known to work well for standard classi- fication, its suitability for fusing rankers has not been studied. In an empirical eval- uation, the resulting approach yields im- proved rankings that are less sensitive to training set variation, and furthermore, the theoretically-motivated reliability indica- tors we introduce enable the metaclassi- fier to now be applicable in any problem where the base classifiers are used. 1 Introduction From business people to the everyday person, e- mail plays an increasingly central role in a modern lifestyle. With this shift, e-mail users desire im- proved tools to help process, search, and organize the information present in their ever-expanding in- boxes. A system that ranks e-mails according to the ∗This work was performed primarily while the first author was supported by Carnegie Mellon University. From: Henry Hutchins <hhutchins@innovative.company.com> To: Sara Smith; Joe Johnson; William Woolings Subject: meeting with prospective customers Hi All, I’d like to remind all of you that the group from GRTY will be visiting us next Friday at 4:30 p.m. The schedule is: + 9:30 a.m. Informal Breakfast and Discussion in Cafeteria + 10:30 a.m. Company Overview + 11:00 a.m. Individual Meetings (Continue Over Lunch) + 2:00 p.m. Tour of Facilities + 3:00 p.m. Sales Pitch In order to have this go off smoothly, I would like to practice the presentation well in advance. As a result, I will need each of your parts by Wednesday. Keep up the good work! –Henry Figure 1: An E-mail with Action-Item (italics added). likelihood of containing “to-do” or action-items can alleviate a user’s time burden and is the subject of ongoing research throughout the literature. In particular, an e-mail user may not always pro- cess all e-mails, but even when one does, some emails are likely to be of greater response urgency than others. These messages often contain action- items. Thus, while importance and urgency are not equal to action-item content, an effective action-item detection system can form one prominent subcom- ponent in a larger prioritization system. Action-item detection differs from standard text classification in two important ways. First, the user is interested both in detecting whether an email contains action-items and in locating exactly where these action-item …

Content truncated to 500 words