Validation and Evaluation of Automatically Acquired Multiword Expressions for Grammar Engineering

Proceedings of the 2007 Joint Conference on Empirical Methods in Natural Language Processing and Computational Natural Language Learning, pp. 1034–1043, Prague, June 2007. c⃝2007 Association for Computational Linguistics Validation and Evaluation of Automatically Acquired Multiword Expressions for Grammar Engineering Aline Villavicencio♣♠, Valia Kordoni♦, Yi Zhang♦, Marco Idiart♥ and Carlos Ramisch♣ ♣Institute of Informatics, Federal University of Rio Grande do Sul (Brazil) ♠Department of Computer Sciences, Bath University (UK) ♦Department of Computational Linguistics, Saarland University, and DFKI GmbH (Germany) ♥Institute of Physics, Federal University of Rio Grande do Sul (Brazil) avillavicencio@inf.ufrgs.br, {yzhang,kordoni}@coli.uni-sb.de idiart@if.ufrgs.br, ceramisch@inf.ufrgs.br Abstract This paper focuses on the evaluation of meth- ods for the automatic acquisition of Multiword Expressions (MWEs) for robust grammar engi- neering. First we investigate the hypothesis that MWEs can be detected by the distinct statistical properties of their component words, regardless of their type, comparing 3 statistical measures: mutual information (MI), χ2 and permutation entropy (PE). Our overall conclusion is that at least two measures, MI and PE, seem to differen- tiate MWEs from non-MWEs. We then investi- gate the influence of the size and quality of differ- ent corpora, using the BNC and the Web search engines Google and Yahoo. We conclude that, in terms of language usage, web generated corpora are fairly similar to more carefully built corpora, like the BNC, indicating that the lack of con- trol and balance of these corpora are probably compensated by their size. Finally, we show a qualitative evaluation of the results of automat- ically adding extracted MWEs to existing lin- guistic resources. We argue that such a process improves qualitatively, if a more compositional approach to grammar/lexicon automated exten- sion is adopted. 1 Introduction The task of automatically identifying Multiword Expressions (MWEs) like phrasal verbs (break down) and compound nouns (coffee machine) using statistical measures has been the focus of considerable investigative effort, (e.g. Pearce (2002), Evert and Krenn (2005) and Zhang et al. (2006)). Given the heterogeneousness of the different phenomena that are considered to be MWEs, there is no consensus about which method is best suited for which type of MWE, and if there is a single method that can be suc- cessfully used for any kind of MWE. Another difficulty for work on MWE identifi- cation is that of the evaluation of the results ob- tained (Pearce, 2002; Evert and Krenn, 2005), starting from the lack of consensus about a pre- cise definition for MWEs (Villavicencio et al., 2005). In this paper we investigate some of the is- sues involved in the evaluation of automatically extracted MWEs, from their extraction to their subsequent use in an NLP task. In order to do that, we present a discussion of different statisti- cal measures, and the influence that the size and quality of different data sources have. We then perform a comparison of these measures and dis- cuss whether there is a single measure that has good overall performance for MWEs in general, regardless of their type. Finally, we perform a qualitative evaluation of the …

Content truncated to 500 words