PEAS, the first instantiation of a comparative framework for evaluating parsers of French

Parsed Corpus Parsed Corpus 2 Participants Transcription Format : XML Xerox / XRCE Grenoble Constitution of a heterogeneous Formated Corpus GREYC / Caen University Tools : 2 specific converters corpus normal i Jai on Parser 1 -> XML 20 K words annotated reference hidden in a 1 million word corpus tokenization sentence segmentation Parser 2 -> XML Scoring Annotated Corpus Metres Precision / Recall Formats : XML / HTML Crossing Brackets Tools. HTML editor 2 specific converters . XML -> HTML HTML -> XML PEAS, the first instantiation of a comparative framework for evaluating parsers of French V. Gendner, G. Illouz, M. Jardino, L. Monceaux, P. Paroubek, I. Robba, A. Vilnat L1MSI — CNRS, BP 133, 91403 Orsay — France {gendner,gabrieli,jardino,monceaux,pap,isabelle,anne}@limsi.fr Abstract This paper presents PEAS, the first comparative evaluation framework for parsers of French whose annotation formalism allows the annotation of both constituents and functional relations. A test corpus containing an assortment of different text types has been built and part of it has been manually annotated. Precision/Recall and crossing brackets metrics will be adapted to our formalism and applied to the parses produced by one parser from academia and another one from industry in order to validate the framework. 1 Introduction In natural language understanding, many complex applications use a syntactic parser as a basic functionality. Today, in particular for the French language, the developers face the great diversity of the offer in the domain. Therefore, the need for a complete comparative evaluation framework — including a pivot annotation formalism, a reference treebank, evaluation metrics and the associated software — is increasing. It is worth noting that most of the recently developed parsers use a robust approach. Consequently, they do not always produce a complete parse of the sentence, but they are able to produce a result, whatever the size, the particularities and the grammaticality of the input. For this reason, it is essential to be able to compare in a fair way the parses they produce against those produced by other parsers whatever their characteristics. One possible solution is to offer a common reference annotation formalism along with a fully parsed reference corpus and a set of robust metrics, allowing for both complete and selective evaluation over an assortment of different text types and syntactic phenomena. The aim of our research is to build such evaluation framework, which to date is missing for French. Figure 1 presents the different modules of our evaluation protocol as it stands today. Figure 1: Evaluation protocol modules 95 Functional relations subject-verb auxiliary-verb argument-verb modifier-verb modifier-noun modifier-adjective modifier-adverb attribute-subject/object Coordination Apposition Complementer 2 Annotation formalism The definition of the annotation formalism is the core element of the evaluation process. Indeed, the formalism must have a coverage of syntactical phenomena as broad as possible in order to allow any parser to participate, whatever the grammatical formalism it uses. We have decided immediately upon a two- steps annotation: first the chunks annotation is carried out, second functional information is annotated through relations between words, …

Content truncated to 500 words