Requests and Commitments in Email are More Complex Than You Think: Eight Reasons to be Cautious

Requests and Commitments in Email are More Complex Than You Think: Eight Reasons to be Cautious Andrew Lampert CSIRO ICT Centre Locked Bag 17 North Ryde 1670 Australia Andrew.Lampert@csiro.au Robert Dale Centre for Language Technology Macquarie University 2109 Australia rdale@ics.mq.edu.au C´ecile Paris CSIRO ICT Centre Locked Bag 17 North Ryde 1670 Australia Cecile.Paris@csiro.au Abstract Many workplace tasks are managed through email communication, involving the exchange of requests and commitments. Our aim is to build a tool that can automatically identify and manage such requests and commitments. A detailed analysis of real data, however, re- veals a range of interesting edge cases that make even human annotation of training data difficult. In this paper, as an important step in the development of annotation guidelines for wider use in the growing email process- ing community, we identify eight categories of problematic data and propose how they should be handled in annotation and extraction tasks. 1 Introduction Our aim is to create tools that assist email users by automatically detecting requests and commitments in incoming and outgoing email. The motivation for this is well explained by observations from ethno- graphic research into the use of electronic messaging in the workplace (Murray, 1991): [Managers] would like to be able to track outstanding promises they have made, promises made to them, requests they’ve made that have not been met and requests made of them that they have not fulfilled. Other studies have also highlighted that people rou- tinely use email for managing requests and commit- ments (e.g., (Mackay, 1988; Ducheneaut and Bel- lotti, 2001)), but struggle to give appropriate atten- tion to requests and commitments that require action or response because they are buried in their email (Whittaker and Sidner, 1996). More recent stud- ies of task-focused email usage have also identified problems with “keeping track of lots of concurrent actions: One’s own to-dos and to-dos one expects from others” using existing email clients (Bellotti et al., 2003). To provide support here, we are working to aug- ment existing email clients with features such as action-oriented summaries of email messages and threads; task-based navigation and visualisations; and dashboards that provide overviews of the state of an email inbox or collection with much greater fidelity than is possible with current tools. In working towards this goal, we have conducted a series of manual annotation experiments, explor- ing the level of human agreement that is achievable in identifying requests and commitments in email messages. Previous work has often relied on canon- ical examples as the basis for simple definitions of requests and commitments. Our experiments have found that this level of detail is insufficient to ob- tain reliable interannotator agreement. Indeed, we have discovered a range of edge cases not well rep- resented by simple examples or definitions. The in- fluence of factors such as politeness lead to a va- riety of more complex and indirect requests and commitments than other researchers have assumed. Such complexity is found when classifying requests and commitments at both the message-level …

Content truncated to 500 words