Dscorer: A Fast Evaluation Metric for Discourse Representation Structure Parsing

Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 4547–4554 July 5 - 10, 2020. c⃝2020 Association for Computational Linguistics 4547 DSCORER: A Fast Evaluation Metric for Discourse Representation Structure Parsing Jiangming Liu Shay B. Cohen Mirella Lapata Institute for Language, Cognition and Computation School of Informatics, University of Edinburgh jiangming.liu@ed.ac.uk, {scohen,mlap}@inf.ed.ac.uk Abstract Discourse representation structures (DRSs) are scoped semantic representations for texts of arbitrary length. Evaluation of the accu- racy of predicted DRSs plays a key role in de- veloping semantic parsers and improving their performance. DRSs are typically visualized as nested boxes, in a way that is not straight- forward to process automatically. COUNTER, an evaluation algorithm for DRSs, transforms them to clauses and measures clause overlap by searching for variable mappings between two DRSs. Unfortunately, COUNTER is com- putationally costly (with respect to memory and CPU time) and does not scale with longer texts. We introduce DSCORER, an efficient new metric which converts box-style DRSs to graphs and then measures the overlap of n-grams in the graphs. Experiments show that DSCORER computes accuracy scores that cor- relate with scores from COUNTER at a fraction of the time. 1 Introduction Discourse Representation Theory (DRT) is a popu- lar theory of meaning representation (Kamp, 1981; Kamp and Reyle, 2013; Asher, 1993; Asher et al., 2003) designed to account for a variety of lin- guistic phenomena within and across sentences. The basic meaning-carrying units in DRT are Dis- course Representation Structures (DRSs). They consist of discourse referents (e.g., x1, x2) rep- resenting entities in the discourse and conditions (e.g., male.n.02(x1), Agent(e1, x1)) representing information about discourse referents. Every vari- able and condition are bounded by a box label (e.g., b1) which implies that the variable or con- dition are interpreted in that box. DRSs are con- structed recursively. An example of a DRS in box- style notation is shown in Figure 1(a). DRS parsing differs from related parsing tasks (e.g., Banarescu et al. 2013) in that it can create rep- resentations that go beyond individual sentences. Despite the large amount of recently developed DRS parsing models (van Noord et al., 2018b; van Noord, 2019; Evang, 2019; Liu et al., 2019b; Fan- cellu et al., 2019; Le et al., 2019), the automatic evaluation of DRSs is not straightforward due to the non-standard DRS format shown in Figure 1(a). It is neither a tree (although a DRS-to-tree conver- sion exists; see Liu et al. 2018, 2019a for details) nor a graph. Evaluation so far relied on COUNTER (van Noord et al., 2018a) which converts DRSs to clauses shown in Figure 1(b). Given two DRSs with n and m (n ≥ m) vari- ables each, COUNTER has to consider n! (n−m)! pos- sible variable mappings in order to find an optimal one for evaluation. The problem of finding this alignment is NP-complete, similar to other metrics such as SMATCH (Cai and Knight, 2013a) for Ab- stract Meaning Representation. COUNTER uses a greedy hill-climbing algorithm to obtain one-to-one variable mappings, …

Content truncated to 500 words