Benchmarking multimedia technologies with the CAMOMILE platform: the case of Multimodal Person Discovery at MediaEval 2015

Benchmarking Multimedia Technologies with the CAMOMILE Platform: the Case of Multimodal Person Discovery at MediaEval 2015 Johann Poignant1, Herv´e Bredin1, Claude Barras1, Mickael Stefas2, Pierrick Bruneau2, Thomas Tamisier2 1. LIMSI, CNRS, Univ. Paris-Sud, Universit´e Paris-Saclay, F-91405 Orsay, firstname.lastname@limsi.fr 2. LIST, Esch-sur-Alzette, Luxembourg, firstname.lastname@list.lu Abstract In this paper, we claim that the CAMOMILE collaborative annotation platform (developed in the framework of the eponymous CHIST-ERA project) eases the organization of multimedia technology benchmarks, automating most of the campaign technical workflow and enabling collaborative (hence faster and cheaper) annotation of the evaluation data. This is demonstrated through the successful organization of a new multimedia task at MediaEval 2015, Multimodal Person Discovery in Broadcast TV. Keywords: evaluation campaign, collaborative annotation, multimedia 1. Introduction For decades, NIST evaluation campaigns have been driving research in the field of human language technology (Mar- tin et al., 2004), recently followed by the CLEF (Peters and Braschler, 2002) and ESTER/ETAPE (Gravier et al., 2004) initiatives. The concept has been successfully transposed to other research areas, such as image recognition (ImageNet Large Scale Visual Recognition Challenge (Russakovsky et al., 2015)), video (TRECVID (Smeaton et al., 2006)) or multimedia indexing (MediaEval (Larson et al., 2015)). More generally, evaluation campaigns allow the assessment of experimental research in fields where human perception and decision must be reproduced by machine learning al- gorithms (Geoffrois, 2008). The general workflow of `a la NIST evaluation campaigns comprises the following stages (Martin et al., 2004): spec- ification of the task; definition of the evaluation metric and provision of an automatic scoring software; design and an- notation of the training, development and evaluation cor- pora; definition of evaluation rules, schedule, protocols and submission formats; sharing of participant results through system descriptions and workshop communications. Automatic scoring is made possible by the manual annota- tion of the data according to the task definition. Costly and time-consuming, this annotation step usually is the main bottleneck of evaluation campaigns. When addressing new tasks in multimodal perception, it becomes challenging (if not impossible) to pre-annotate the ever-increasing volume of multimedia data. A compromise has been successfully explored in the TREC and TRECVid campaigns, where the annotation of a small (but carefully chosen (Yilmaz and Aslam, 2006)) subset of the test data is bootstrapped by the participants’ submissions. In this paper, we claim that the CAMOMILE collabora- tive annotation platform (developed in the framework of the eponymous CHIST-ERA project) eases the organization of multimedia technology benchmarks, automating most of the campaign technical workflow and enabling collabora- tive (hence faster and cheaper) annotation of the evaluation data. This is demonstrated through the successful organi- zation of a new multimedia task at MediaEval 2015, Multi- modal Person Discovery in Broadcast TV (Poignant et al., 2015b). 2. Multimodal Person Discovery in Broadcast TV The objective of this new task is to make TV archives fully exploitable and searchable through people indexing. Par- ticipants were provided with a collection of TV broadcast recordings pre-segmented into shots. Each shot had to be automatically tagged with the names of …

Content truncated to 500 words