UP13: Knowledge-poor Methods (Sometimes) Perform Poorly

Proceedings of the 4th International Workshop on Semantic Evaluations (SemEval-2007), pages 418–421, Prague, June 2007. c⃝2007 Association for Computational Linguistics UP13: Knowledge-poor Methods (Sometimes) Perform Poorly Thierry Poibeau Laboratoire d’Informatique de Paris-Nord CNRS UMR 7030 et université Paris 13 99, avenue J.-B. Clément F-93430 Villetaneuse thierry.poibeau@lipn.univ-paris13.fr Abstract This short paper presents a system developed at the Université Paris 13 for the Semeval 2007 Metonymy Resolution Task (task #08, location name track; see Markert and Nissim, 2007). The system makes use of plain word forms only. In this paper, we evaluate the accuracy of this minimalist approach, compare it to a more complex one which uses both syntactic and semantic features, and discuss its usefulness for metonymy resolution in general. 1 Introduction This short paper presents the system developed at the Université Paris 13 for the Metonymy resolution task, during Semeval 2007 (Markert and Nissim, 2007). Two sub-tasks were proposed, concerning 1) country names and 2) company names. We only participated in the first task (country names). We developed a simple approach which we present and thoroughly evaluate in this paper. We discuss the relevance of this approach and compare it to more complex ones. 2 Motivation We participated in the metonymy task with a very basic system. The idea was to investigate the efficiency of a minimalist (though, not Chomskian) system. This system tags entities on the basis of discriminative (plain) word forms occurring in a given window only. Our aim was to find out which word forms are discriminative enough to be considered as parameters. In the past, we developed a system for metonymy resolution for French, evaluated in the framework of the ESTER evaluation (Gravier, 2004). This system, described in Poibeau (2006), uses various kinds of information, among others: plain word forms, part-of-speech tags, and syntactic and semantic tags (conceptual word classes). The usefulness of complex linguistic features (especially syntactic and semantic tags) is questionable: they may be hard to compute, error- prone and their contribution is not clear. We therefore developed a new version of the system mainly based on 1) a distributional analysis (on surface word forms) along with 2) a filtering process. The latter restricted metonymic readings to country and capital names (as opposed to other location names), since they include a vast majority of the metonymic readings (this proved to be efficient but is of course a harsh pragmatic over- simplification without real linguistic basis). We nevertheless obtained a highly versatile system, performing reasonably well, compared to our previous, much more complex implementation (F-score was .58 instead of .63; we computed F-score with β=1). In the framework of the Semeval evaluation, the filtering process is irrelevant since only country names are considered as entities. However, we thought that it would be interesting to develop a very basic system, to evaluate the performance one can obtain using plain word forms only. 3 A (too) Lazy Approach We chose not to use any part-of-speech tagger or syntactic or semantic analyzer; we did not use any external …

Content truncated to 500 words