You Can’t Beat Frequency (Unless You Use Linguistic Knowledge) – A Qualitative Evaluation of Association Measures for Collocation and Term Extraction

Proceedings of the 21st International Conference on Computational Linguistics and 44th Annual Meeting of the ACL, pages 785–792, Sydney, July 2006. c⃝2006 Association for Computational Linguistics You Can’t Beat Frequency (Unless You Use Linguistic Knowledge) – A Qualitative Evaluation of Association Measures for Collocation and Term Extraction Joachim Wermter Udo Hahn Jena University Language & Information Engineering (JULIE) Lab D-07743 Jena, Germany {wermter|hahn}@coling-uni-jena.de Abstract In the past years, a number of lexical association measures have been studied to help extract new scientific terminol- ogy or general-language collocations. The implicit assumption of this research was that newly designed term measures involv- ing more sophisticated statistical criteria would outperform simple counts of co- occurrence frequencies. We here explic- itly test this assumption. By way of four qualitative criteria, we show that purely statistics-based measures reveal virtually no difference compared with frequency of occurrence counts, while linguistically more informed metrics do reveal such a marked difference. 1 Introduction Research on domain-specific automatic term recognition (ATR) and on general-language collo- cation extraction (CE) has gone mostly separate ways in the last decade although their underlying procedures and goals turn out to be rather simi- lar. In both cases, linguistic filters (POS taggers, phrase chunkers, (shallow) parsers) initially col- lect candidates from large text corpora and then frequency- or statistics-based evidence or associa- tion measures yield scores indicating to what de- gree a candidate qualifies as a term or a colloca- tion. While term mining and collocation mining, as a whole, involve almost the same analytical pro- cessing steps, such as orthographic and morpho- logical normalization, normalization of term or collocation variation etc., it is exactly the measure which grades termhood or collocativity of a can- didate on which alternative approaches diverge. Still, the output of such mining algorithms look similar. It is typically constituted by a ranked list on which, ideally, the true terms or collocations are placed in the top portion of the list, while the non-terms / non-collocations occur in its bottom portion. While there have been lots of approaches to come up with a fully adequate ATR/CE metric (cf. Section 2), we have made observations in our experiments that seem to indicate that simplicity rules, i.e., frequency of occurrence is the dominat- ing factor for the ranking in the result lists even when much smarter statistical machinery is em- ployed. In this paper, we will discuss data which reveals that purely statistics-based measures ex- hibit virtually no difference compared with fre- quency of occurrence counts, while linguistically more informed measures do reveal such a marked difference – for the problem of term and colloca- tion mining at least. 2 Related Work Although there has been a fair amount of work employing linguistically sophisticated analysis of candidate items (e.g., on CE by Lin (1998) and Lin (1999) as well as on ATR by Daille (1996), Jacquemin (1999), and Jacquemin (2001)), these approaches are limited by the difficulty to port grammatical specifications to other domains (in the case of ATR) or by the …

Content truncated to 500 words