DocAMR: Multi-Sentence AMR Representation and Evaluation

Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 3496 - 3505 July 10-15, 2022 ©2022 Association for Computational Linguistics DOCAMR: Multi-Sentence AMR Representation and Evaluation Tahira Naseem♣,∗ Austin Blodgett♠ Sadhana Kumaravel♣ Tim O’Gorman♡ Young-Suk Lee♣ Jeffrey Flanigan♢ Ramón Fernandez Astudillo♣ Radu Florian♣ Salim Roukos♣ Nathan Schneider△ ∗tnaseem@us.ibm.com ♣IBM Research AI ♠U.S. Army Research Laboratory, Adelphi, MD ♡Thorn ♢University of California, Santa Cruz △Georgetown University Abstract Despite extensive research on parsing of En- glish sentences into Abstract Meaning Repre- sentation (AMR) graphs, full-document parsing into a unified graph representation lacks well- defined representation and evaluation. Taking advantage of a super-sentential level of AMR coreference annotation from previous work, we introduce a simple algorithm for deriving a uni- fied graph representation, avoiding the pitfalls of information loss from over-merging and lack of coherence from under-merging. Next, we describe improvements to the Smatch metric to make it tractable for comparing document-level graphs, and use it to re-evaluate the best pub- lished document-level AMR parser. We also present a pipeline approach combining the top performing AMR parser and coreference reso- lution systems, providing a strong baseline for future research. 1 Introduction Abstract Meaning Representation (AMR) is a for- malism that represents the meaning of a text in the form of a directed graph, where nodes represent concepts and edges are labeled with relations (Ba- narescu et al., 2013). Until recently, the annotated corpora for AMR only provided sentence-level an- notations. As a result, AMR parsing research has been limited to sentence-level parsing. A unified document-level AMR graph representation can be quite useful for applications that require document- level understanding, such as question answering and summarization. The most recent release of AMR annotations (AMR 3.0) includes a multi-sentence AMR cor- pus (MS-AMR; O’Gorman et al., 2018) that pro- vides cross-sentential coreference information for AMR graphs. These annotations connect multiple sentence-level graphs via coreference chains, im- plicit relations and bridging relations. However, an adequate representation of this multi-sentence in- formation in the form of an AMR graph has yet to be decided upon. The initial proposal (O’Gorman et al., 2018), to merge all nodes in a coreference chain into one node, suffers from significant infor- mation loss. Without a consistent multi-sentence graph representation, AMR’s standard Smatch met- ric (Cai and Knight, 2013) cannot be used for eval- uation and comparison. The Smatch scores differ greatly depending on how the coreference infor- mation is added to the AMR graphs. Additionally, the Smatch algorithm is exceedingly slow in its original form when run over multi-sentence graphs. These limitations in multi-sentence AMR graph representation and evaluation get in the way of re- search efforts in this area. Without a consistent representation and evaluation mechanism, it is im- possible to draw meaningful comparisons between different approaches on the task. In this work, we present a simple and non-lossy multi-sentence AMR graph representation as well as a modifica- tion of the Smatch algorithm that allows efficient and consistent evaluation.1 …

Content truncated to 500 words