Methods for the Qualitative Evaluation of Lexical Association Measures

Methods for the Qualitative Evaluation of Lexical Association Measures Stefan Evert IMS, University of Stuttgart Azenbergstr. 12 D-70174 Stuttgart, Germany evert@ims.uni-stuttgart.de Brigitte Krenn Austrian Research Institute for Artificial Intelligence (ÖFAI) Schottengasse 3 A-1010 Vienna, Austria brigitte@ai.univie.ac.at Abstract This paper presents methods for a qual- itative, unbiased comparison of lexical association measures and the results we have obtained for adjective-noun pairs and preposition-noun-verb triples ex- tracted from German corpora. In our approach, we compare the entire list of candidates, sorted according to the particular measures, to a reference set of manually identified “true positives”. We also show how estimates for the very large number of hapaxlegomena and double occurrences can be inferred from random samples. 1 Introduction In computational linguistics, a variety of (statis- tical) measures have been proposed for identify- ing lexical associations between words in lexi- cal tuples extracted from text corpora. Methods used range from pure frequency counts to infor- mation theoretic measures and statistical signifi- cance tests. While the mathematical properties of those measures have been extensively discussed,1 the strategies employed for evaluating the iden- tification results are far from adequate. Another crucial but still unsolved issue in statistical col- location identification is the treatment of low- frequency data. In this paper, we first specify requirements for a qualitative evaluation of lexical association mea- 1See for instance (Manning and Schütze, 1999, chap- ter 5), (Kilgarriff, 1996), and (Pedersen, 1996). sures (AMs). Based on these requirements, we introduce an experimentation procedure, and dis- cuss the evaluation results for a number of widely used AMs. Finally, methods and strategies for handling low-frequency data are suggested. The measures2 – Mutual Information ( �� ) (Church and Hanks, 1989), the log-likelihood ratio test (Dunning, 1993), two statistical tests: t-test and �� -test, and co-occurrence frequency – are applied to two sets of data: adjective-noun (AdjN) pairs and preposition-noun-verb (PNV) triples, where the AMs are applied to (PN,V) pairs. See section 3 for a description of the base data. For evaluation of the association measures, � -best strategies (section 4.1) are supplemented with precision and recall graphs (section 4.2) over the complete data sets. Samples comprising par- ticular frequency strata (high versus low frequen- cies) are examined (section 4.3). In section 5, methods for the treatment of low-frequency data, single (hapaxlegomena) and double occurrences are discussed. The significance of differences be- tween the AMs is addressed in section 6. 2 The Qualitative Evaluation of Association Measures 2.1 State-of-the-art A standard procedure for the evaluation of AMs is manual judgment of the � -best candidates identi- fied in a particular corpus by the measure in ques- tion. Typically, the number of true positives (TPs) 2For a more detailed description of these measures and relevant literature, see (Manning and Schütze, 1999, chapter 5) or http://www.collocations.de/EK/, where several other AMs are discussed as well. among the 50 or 100 (or slightly more) highest ranked word combinations is manually identified by a human evaluator, in most cases the author of the paper in which the evaluation …

Content truncated to 500 words