EMMA: A novel Evaluation Metric for Morphological Analysis

Proceedings of the 23rd International Conference on Computational Linguistics (Coling 2010), pages 1029–1037, Beijing, August 2010 EMMA: A Novel Evaluation Metric for Morphological Analysis Sebastian Spiegler Intelligent Systems Group University of Bristol spiegler@cs.bris.ac.uk Christian Monson Center for Spoken Language Understanding Oregon Health & Science University monsonc@csee.ogi.edu Abstract We present a novel Evaluation Metric for Morphological Analysis (EMMA) that is both linguistically appealing and empirically sound. EMMA uses a graph- based assignment algorithm, optimized via integer linear programming, to match morphemes of predicted word analyses to the analyses of a morphologically rich answer key. This is necessary especially for unsupervised morphology analysis systems which do not have access to linguistically motivated morpheme labels. Across 3 languages, EMMA scores of 14 systems have a substantially greater positive correlation with mean average precision in an information retrieval (IR) task than do scores from the metric currently used by the Morpho Challenge (MC) competition series. We compute EMMA and MC metric scores for 93 separate system-language pairs from the 2007, 2008, and 2009 MC compe- titions, demonstrating that EMMA is not susceptible to two types of gaming that have plagued recent MC competi- tions: Ambiguity Hijacking and Shared Morpheme Padding. The EMMA eval- uation script is publicly available from http://www.cs.bris.ac.uk/ Research/MachineLearning/ Morphology/Resources/. 1 Introduction Words in natural language are constructed from smaller building blocks called morphemes. For example, the word wives breaks down into an un- derlying stem, wife, together with a plural suffix. Analyzing the morphological structure of words is known to benefit a variety of downstream nat- ural language (NL) tasks such as speech recogni- tion (Creutz, 2006; Arısoy et al., 2009), machine translation (Oflazer et al., 2007), and information retrieval (McNamee et al., 2008). A variety of automatic systems can morpholog- ically analyze words that have been removed from their surrounding context. These systems range from hand-built finite state approaches (Beesley and Karttunen, 2003) to recently proposed algo- rithms which learn morphological structure in an unsupervised fashion (Kurimo et al., 2007). Since unsupervised systems do not have access to lin- guistically motivated morpheme labels, they typ- ically produce morphological analyses that are closely related to the written form. Such a system might decompose wives as wiv -es. Meanwhile, a hand-built system might propose wife_N +Plu- ral, or even parse wives as a hierarchical feature structure. As morphological analysis systems pro- duce such varied outputs, comparing decomposi- tions from disparate systems is a challenge. This paper describes EMMA, an Evaluation Metric for Morphological Analysis that quantita- tively measures the quality of a set of morpholog- ical analyses in a linguistically adequate, empir- ically useful, and novel fashion. EMMA evalu- ates analyses that can be represented as a flat set of symbolic features, including hierarchical repre- sentations, which can be projected down to a lin- earized form (Roark and Sproat, 2007). An automatic metric that discriminates be- tween proposed morphological analyses should 1029 fulfill certain computational and linguistic crite- ria. Computationally, the metric should: 1. Correlate with the performance of real-world NL processing …

Content truncated to 500 words