When Beards Start Shaving Men: A Subject-object Resolution Test Suite for Morpho-syntactic and Semantic Model Introspection

Proceedings of the 28th International Conference on Computational Linguistics, pages 3019–3035 Barcelona, Spain (Online), December 8-13, 2020 3019 When Beards Start Shaving Men: A Subject-object Resolution Test Suite for Morpho-syntactic and Semantic Model Introspection Patricia Fischer and Dani¨el de Kok and Erhard Hinrichs University of T¨ubingen {patricia.fischer, daniel.de-kok, erhard.hinrichs} @uni-tuebingen.de Abstract In this paper, we introduce the SORTS Subject-Object Resolution Test Suite of German minimal sentence pairs for model introspection.1 The full test suite consists of 18,502 transitive clauses with manual annotations of 8 word order patterns, 5 morphological and syntactic and 11 semantic property classes. The test suite has been constructed such that sentences are minimal pairs with respect to a property class. Each property has been selected with a particular focus on its effect on subject-object resolution, the second-most error-prone task within syntactic parsing of German after prepositional phrase attachment (Fischer et al., 2019). The size and detail of annotations make the test suite a valuable resource for natural language processing applications with syntactic and semantic tasks. We use dependency parsing to demonstrate how the test suite allows insights into the process of subject-object resolution. Based on the test suite annotations, word order and case syncretism can be identified as most important factors that affect subject-object resolution. 1 Introduction Subject-object resolution remains a difficult task for syntactic parsing of languages with relatively free word order and case syncretism. For such languages, subject and object cannot always be disambiguated based on morpho-syntactic surface structures (Lenerz, 1977b; Eisenberg, 2013). Parser performance on German and Dutch, for example, has been shown to suffer significantly from incorrect identification of subject and object (Van Noord, 2007; Fischer et al., 2019). While there have been task-specific test suites for difficult syntactic phenomena in German such as prepositional phrase attachment, coordination, and verb phrase complementation (Nerbonne et al., 1991; Lehmann et al., 1996; K¨ubler et al., 2009), subject- object resolution has widely been neglected in existing test suites. In this paper, we introduce the SORTS Subject-Object Resolution Test Suite of German minimal sen- tence pairs. The test suite has been created manually from hand-selected subject-verb-object triples and contains 18,502 transitive clauses with Universal Dependency (UD, De Marneffe et al. (2014)) annota- tions, template-based annotations of 8 word order patterns and manual annotations of 16 morphological, syntactic and semantic property classes. For instance, the sentence Sie trifft eine Entscheidung. ‘She makes a decision.’ has been annotated with word order VF[S]LK[V]MF[O], the syntactic property subject pronoun and the semantic property light verb in addition to dependency relations rel- evant for subject-object resolution, as illustrated in Figure 1. One domain of application for this test suite is syntactic parsing. We will use six neural dependency parsers with different architectures to show that the test suite is able to expose the linguistic properties that make subject-object resolution easier or more difficult for parsing. Experimental results imply that parsers are not able to resolve certain syntactic structures well, e.g. subject-object pairs with object- subject order and case syncretism. The paper …

Content truncated to 500 words