Evaluating Scoped Meaning Representations

Evaluating Scoped Meaning Representations Rik van Noord, Lasha Abzianidze, Hessel Haagsma, Johan Bos CLCG, University of Groningen {r.i.k.van.noord, l.abzianidze, hessel.haagsma, johan.bos}@rug.nl Abstract Semantic parsing offers many opportunities to improve natural language understanding. We present a semantically annotated parallel corpus for English, German, Italian, and Dutch where sentences are aligned with scoped meaning representations in order to capture the semantics of negation, modals, quantification, and presupposition triggers. The semantic formalism is based on Discourse Representation Theory, but concepts are represented by WordNet synsets and thematic roles by VerbNet relations. Translating scoped meaning representations to sets of clauses enables us to compare them for the purpose of semantic parser evaluation and checking translations. This is done by computing precision and recall on matching clauses, in a similar way as is done for Abstract Meaning Representations. We show that our matching tool for evaluating scoped meaning representations is both accurate and efficient. Applying this matching tool to three baseline semantic parsers yields F-scores between 43% and 54%. A pilot study is performed to automatically find changes in meaning by comparing meaning representations of translations. This comparison turns out to be an additional way of (i) finding annotation mistakes and (ii) finding instances where our semantic analysis needs to be improved. Keywords: parallel corpus, semantic annotation, discourse representation structure, evaluation, semantic scope 1. Introduction Semantic parsing is the task of assigning meaning represen- tations to natural language expressions. Informally speak- ing, a meaning representation describes who did what to whom, when, and where, and to what extent this is the case or not. The availability of open-domain, wide coverage se- mantic parsers has the potential to add new functionality, such as detecting contradictions, verifying translations, and getting more accurate search results. Current research on open-domain semantic parsing focuses on supervised learn- ing methods, using large semantically annotated corpora as training data. However, there are not many annotated corpora avail- able. We present a parallel corpus annotated with for- mal meaning representations for English, Dutch, German, and Italian, and a way to evaluate the quality of machine- generated meaning representations by comparing them to gold standard annotations. Our work shows many simi- larities with recent annotation and parsing efforts around Abstract Meaning Representations, (AMR; Banarescu et al., 2013) in that we abstract away from syntax, use first- order meaning representations, and use an adapted version of SMATCH (Cai and Knight, 2013) for evaluation. How- ever, we deviate from AMR on several points: meanings are represented by scoped meaning representations (arriv- ing at a more linguistically motivated treatment of modals, negation, presupposition, and quantification), and the non- logical symbols that we use are grounded in WordNet (con- cepts) and VerbNet (thematic roles), rather than PropBank (Palmer et al., 2005). We also provide a syntactic analy- sis in the annotated corpus, in order to derive the semantic analyses in a compositional way. We make the following contributions: • A meaning representation with explicit scopes that combines WordNet and VerbNet with elements of for- mal logic (Section 2). …

Content truncated to 500 words