Butterfly Effects in Frame Semantic Parsing: impact of data processing on model ranking

Proceedings of the 27th International Conference on Computational Linguistics, pages 3158–3169 Santa Fe, New Mexico, USA, August 20-26, 2018. 3158 Butterfly Effects in Frame Semantic Parsing: impact of data processing on model ranking Alexandre Kabbach Department of Linguistics University of Geneva Corentin Ribeyre Etermind {firstname.lastname}@{unige.ch;etermind.com;unitn.it} Aur´elie Herbelot Center for Mind/Brain Sciences & Dept. of Information Engineering and Computer Science University of Trento Abstract Knowing the state-of-the-art for a particular task is an essential component of any computational linguistics investigation. But can we be truly confident that the current state-of-the-art is indeed the best performing model? In this paper, we study the case of frame semantic parsing, a well- established task with multiple shared datasets. We show that in spite of all the care taken to provide a standard evaluation resource, small variations in data processing can have dramatic consequences for ranking parser performance. This leads us to propose an open-source standard- ized processing pipeline, which can be shared and reused for robust model comparison. Title and Abstract in French Effets papillon en analyse automatique de cadres s´emantiques: impact du traitement des donn´ees sur la hi´erarchie des mod`eles Connaˆıtre l’´etat-de-l’art pour une tˆache donn´ee est un ´el´ement essentiel de toute entreprise de linguistique computationnelle. Peut-on n´eanmoins s’assurer que l’´etat-de-l’art connu est bel et bien le meilleur des mod`eles ? Dans ces travaux nous nous int´eressons au cas de l’analyse automatique de cadres s´emantiques, une tˆache bien ´etablie disposant de multiples jeux de donn´ees partag´es. Nous montrons qu’en d´epit du soin port´e `a la standardisation du processus d’´evaluation, de faibles variations dans le pr´etraitement des donn´ees peuvent conduire `a une remise en question du classement final des analyseurs. Cette constatation nous conduit `a proposer une plateforme libre de traitement standardis´ee permettant d’obtenir des donn´ees de comparaisons fiables entre les performances des diff´erents analyseurs. 1 Introduction A typical contribution to Computational Linguistics research is a new model for a specific task, which improves performance on a known dataset. The proposed model, whilst being the object of focus in the associated publication, normally relies on some potentially extensive pre- and post-processing of data which may only be briefly reported. As shown in the seminal work by (Fokkens et al., 2013), small alterations to such processing can result in dramatic changes in model performance. In this paper, we pick up on this point and make a case for harmonizing evaluations by sharing not just datasets but entire experimental pipelines. We illustrate our point by focusing on recent frame semantic parsing literature. Frame semantic parsing is the task of automatically extracting semantic structures in text following the theory of Frame Semantics (Fillmore, 1982) and the framework of FrameNet (Baker et al., 1998, hereafter FN). Formally defined in the SemEval 2007 shared task 19 (Baker et al., 2007), it consists of three separate subtasks: (1) target identification: the task of identifying all frame evoking words in a given sentence; (2) frame identification: the task of identifying all frames of pre-identified targets in a given sentence; and (3) …

Content truncated to 500 words