Matics Software Suite: New Tools for Evaluation and Data Exploration

Matics Software Suite: New Tools for Evaluation and Data Exploration Olivier Galibert, Guillaume Bernard, Agnes Delaborde, Sabrina Lecadre, Juliette Kahn Laboratoire national de m´etrologie et d’essais 1 rue Gaston Boissier - 75015 Paris, France firstname.lastname@lne.fr Abstract Matics is a free and open-source software suite for exploring annotated data and evaluation results. It proposes a dataframe data model allowing the intuitive exploration of data characteristics and evaluation results and provides support for graphing the values and running appropriate statistical tests. The tools already run on several Natural Language Processing tasks and standard annotation formats, and are under on-going development. 1. Introduction The evaluation of data processing systems is a cornerstone for developers, researchers and users. The evaluation al- lows positioning a technology with regard to the compe- tition, but also allows assessing the performance of the system in different contexts. Through quantified scores, it orientates the development or guides the user towards the most suitable product. As evidenced by the popular- ity of evaluation competitions such as the VarDial eval- uation campaigns (Malmasi et al., 2016), the Interspeech challenges (Schuller et al., 2017), the CoNLL Shared Tasks (Oepen et al., 2017) or the Evalita evaluation cam- paign (Pierpaolo et al., 2017), there is a constant need for the rapidly-evolving Natural Language Processing (NLP) technologies to position themselves. The evaluation of text, speech or multimedia data pro- cessing systems relies on large amounts of usually anno- tated data. Numerous works propose interfaces or frame- works to build, explore and visualize corpora of annotated data. ANVIL (Kipp, 2010) for instance proposes a well- conceived database-oriented annotation tool where the user can add temporally or spatially grounded elements. The an- notated data can be exported to perform statistical analyses in several external systems. UAM (O’Donnell, 2008) em- phasizes the project management aspects in a multi-layer text annotation task and offers dedicated statistical analysis tools. Headtalk/Handtalk (Knight et al., 2009) explores the annotation and visualization of multimodal corpora, for the purpose of building datasets suitable for statistic analysis. Some propose a framework able to explore data in a spe- cific context. (Schmitt et al., 2010) for instance is dialogue- oriented, and presents a multi-level interface including di- alog selection from a database, display of the selected dia- log, and application and evaluation of integrated prediction models for various characteristics (task completion, anger level, age and gender predictions). All these systems are mainly dedicated to annotation tasks and/or to specific NLP applications. They offer data ex- ploration features, and can either produce data formatted to perform an evaluation in an external system, or offer statis- tical analysis specialized for testing coherency or measur- ing advance on a specific NLP task. The LNE (laboratoire national de m´etrologie et d’essais - French national metrology and testing laboratory) has conducted many evaluations of data-processing systems in projects such as Quaero (Galibert et al., 2011), ETAPE (Gravier et al., 2012), MAURDOR (Brunessaux et al., 2014), PEA-TRAD or REPERE (Giraudel et al., 2012; Galibert and Kahn, 2013). These evaluations con- cerned …

Content truncated to 500 words