Evaluation Strategies for Computational Construction Grammars

Proceedings of COLING 2016, the 26th International Conference on Computational Linguistics: Technical Papers, pages 1137–1146, Osaka, Japan, December 11-17 2016. Evaluation Strategies for Computational Construction Grammars Tˆania Marques School of Informatics University of Edinburgh Edinburgh EH8 9AB United Kingdom tmarques@inf.ed.ac.uk Katrien Beuls Artificial Intelligence Lab Vrije Universiteit Brussel Pleinlaan 2 B-1050 Brussels Belgium katrien@ai.vub.ac.be Abstract Despite the growing number of Computational Construction Grammar implementations, the field is still lacking evaluation methods to compare grammar fragments across different platforms. Moreover, the hand-crafted nature of most grammars requires profiling tools to understand the complex interactions between constructions of different types. This paper presents a number of evaluation measures, partially based on existing measures in the field of semantic parsing, that are especially relevant for reversible grammar formalisms. The measures are tested on a grammar fragment for European Portuguese clitic placement that is currently under development. 1 Introduction Computational Construction Grammar allows computational linguists to formalize their hypotheses and intuitions about certain linguistic phenomena and explore how these representational choices affect the processing of natural language utterances (Schneider and Tsarfaty, 2013). In this sense, it follows in the footsteps of Generalized Phrase Structure Grammar (GPSG) (Gazdar et al., 1985), Lexical Functional Grammar (Bresnan et al., 2016), Head-Driven Phrase-Structure Grammar (HPSG) (Pollard and Sag, 1994) and Combinatory Categorial Grammar (CCG) (Steedman, 2000). Yet, different from these other approaches, it adheres to the principles of Construction Grammar (CxG) (Goldberg, 2005; ¨Ostman and Fried, 2005; Hoffmann and Trousdale, 2013).Therefore, constructions are treated as first-class citizens in the grammatical organisation of a language. They are viewed as learned mappings between form (sounds, morphemes, syntactic categories) and function (semantics, pragmatics, etc.). Because there is no strict separation between the lexicon and the grammar, semi-productive idioms like “X let alone Y” are treated in the same way with lexemes and core syntactic patterns (Fillmore et al., 1988). We can distinguish three main computational frameworks that are currently active within the Construction Grammar community: Embodied Construction Grammar – ECG (Bergen and Chang, 2005), Fluid Construction Grammar – FCG (Steels, 2004; Steels, 2011) and more recently Template Construction Grammar – TCG (Barr`es and Lee, 2014). The evaluation of computational construction grammars is currently not reaching further than proof- of-concept grammar fragments that show how to implement a certain language phenomenon and demon- strate the resulting grammar by means of web demonstrations or its use in a simulation-based robotic environment (Trott et al., 2015). There are two reasons for the lack of widely used evaluation metrics in CxG: (i) Different from data-driven approaches, construction grammars are not built automatically from annotated treebanks and therefore do not reach a wide coverage in the traditional sense. Instead, both ECG and FCG allow the grammar writer to test a number of sentences automatically when loading the grammar fragment and return the average base parse for these (geometric mean of the number of parses per sentence). (ii) Different from syntactic parsers that concentrate on the syntactic accuracy of the syn- tax trees that their grammars derive, …

Content truncated to 500 words