A Tale of Three Parsers: Towards Diagnostic Evaluation for Meaning Representation Parsing

Proceedings of the 12th Conference on Language Resources and Evaluation (LREC 2020), pages 1902–1909 Marseille, 11–16 May 2020 c⃝ European Language Resources Association (ELRA), licensed under CC-BY-NC 1902 A Tale of Three Parsers†: Towards Diagnostic Evaluation for Meaning Representation Parsing Maja Buljan♣, Joakim Nivre♠, Stephan Oepen♣, and Lilja Øvrelid♣ ♣ University of Oslo, Department of Informatics ♠ Uppsala University, Department of Linguistics and Philology { majabu | oe | liljao } @ifi.uio.no , joakim.nivre@lingfil.uu.se Abstract We discuss methodological choices in contrastive and diagnostic evaluation in meaning representation parsing, i.e. mapping from natural language utterances to graph-based encodings of semantic structure. Drawing inspiration from earlier work in syntactic dependency parsing, we transfer and refine several quantitative diagnosis techniques for use in the context of the 2019 shared task on Meaning Representation Parsing (MRP). As in parsing proper, moving evaluation from simple rooted trees to general graphs brings along its own range of challenges. Specifically, we seek to begin to shed light on relative strenghts and weaknesses in different broad families of parsing techniques. In addition to these theoretical reflections, we conduct a pilot experiment on a selection of top-performing MRP systems and two of the five meaning representation frameworks in the shared task. Empirical results suggest that the proposed methodology can be meaningfully applied to parsing into graph-structured target representations, uncovering hitherto unknown properties of the different systems that can inform future development and cross-fertilization across approaches. Keywords: Data-Driven Parsing, Sentence Semantics, Meaning Representation Parsing, Contrastive Evaluation, Diagnostics 1. Introduction In no small part following on from decades of progress in (surface) syntactic parsing, there is now growing interest in parsing into ‘deeper’ and more abstract representations of sentence structure. In particular, encoding sentence mean- ing in the form of labeled directed graphs has been the focus of a series of annual parsing competitions since the first shared task on Semantic Dependency Parsing (SDP) at the 2014 Workshop on Semantic Evaluation (Oepen et al., 2014). Adopting the perspective of the most recent such shared task, at the 2019 Conference on Computational Nat- ural Language Learning (CoNLL), we refer to this line of research as meaning representation parsing (Oepen et al., 2019). While most representations of syntactic structure limit themselves to rooted trees, common frameworks for mean- ing representation assume general graphs. These structures make the parsing task much more complex—often mov- ing from techniques with polynomial worst-case complex- ity to problems that are in principle NP-hard. Among other things, meaning representations transcend syntactic trees in allowing nodes with in-degree greater than one (‘reentran- cies’), multiple roots, and ignoring semantically ‘vacuous’ parts of the parser input. Besides greatly increased mod- eling and algorithmic complexity, meaning representation parsing also poses its own set of methodological challenges for parser evaluation and diagnostics. The contrastive studies initiated by McDonald and Nivre (2007) and McDonald and Nivre (2011) have been influ- ential in comparing the performance of two core types of approaches to syntactic dependency parsing, i.e. different families of parsing approaches. In this work, we …

Content truncated to 500 words