Summarizing Behaviours: An Experiment on the Annotation of Call-Centre Conversations

Summarizing Behaviour: An Experiment on the Annotation of Call-Centre Conversations Morena Danieli1, Balamurali A R2, Evgeny A. Stepanov1 Benoit Favre2, Frederic Bechet2, Giuseppe Riccardi1 1Signals and Interactive Systems Lab, University of Trento, Trento, Italy 2Aix-Marseille Université, CNRS, LIF UMR 7279, 13000, Marseille, France Corresponding author: morena.danieli@unitn.it Abstract Annotating and predicting behavioural aspects in conversations is becoming critical in the conversational analytics industry. In this paper we look into inter-annotator agreement of agent behaviour dimensions on two call center corpora. We find that the task can be annotated consistently over time, but that subjectivity issues impacts the quality of the annotation. The reformulation of some of the annotated dimensions is suggested in order to improve agreement. Keywords: behavioural analytics, corpora annotation, inter-annotator agreement 1. Introduction One of the most critical problems of contact centre business is the quality assessment of the huge amount of inbound and outbound phone calls they receive every day. Calls between customers and agents need to meet several quality requirements that include both objective measures, like first call resolution and monitoring of caller waiting time, and more intangible criteria related with the communicative appropriateness of the agents. Some contact center companies constantly evaluate the quality of the calls by applying observational methods such as qualified listening of randomly selected conversations. While those methods present some advantages, they are highly subjective, and run the risk of being statistically invalid since the traditional observational methods often target only 2 to 10 calls (or interactions) per agent per month. In the SENSEI project we design, develop, and evaluate methods and tools based on analytics technologies that may be helpful for approaching the problem described above. The goal of SENSEI is to provide different types of summaries of huge amount of spoken and written conversations, including call center conversations and social media interactions. In this paper we focus on the evaluation of the behavioural annotation of call center conversations. 2. The Annotation Task In a real call centre the live conversations are assessed by Quality Assurance (QA henceforth) supervisors and are scored against established contact handling criteria, summarised into a QA questionnaire. The typical survey asks questions like “Was the agent able to provide the client with the listening to the calls, the QA supervisor judges if the agent passes of fails the criterion. Most of the questions are focused on the behavioural attitude of the agent, and replying to them is a highly subjective task. Starting from the analysis of actually used Agent Conversation Observation Forms (ACOF), we developed observation surveys that were applied to two corpora of phone conversations, the Italian corpus LUNA and the French corpus RATP-DECODA. Both corpora are collection of inbound calls, but they come from different semantic domains: the LUNA conversations are calls to technical assistance services by corporate users, the DECODA ones are information inquiry calls by generic users. This difference implies different requirements for the work of call center agents, so it was necessary to define for each corpus a specific questionnaire with …

Content truncated to 500 words