Evaluating Diversity of Multiword Expressions in Annotated Text

Proceedings of the 29th International Conference on Computational Linguistics, pages 3285–3295 October 12–17, 2022. 3285 Evaluating Diversity of Multiword Expressions in Annotated Text Adam Lion-Bouton1, Ya˘gmur Öztürk2, Agata Savary2, Jean-Yves Antoine1 University of Tours - LIFAT1, Paris-Saclay University, CNRS - LISN2, {lion.adam.otman, yaamurozturk}@gmail.com agata.savary@universite-paris-saclay.fr jean-yves.antoine@univ-tours.fr Abstract Diversity can be decomposed into three dis- tinct concepts, namely: variety, balance and dis- parity. This paper borrows from the extensive formalization and measures of diversity devel- oped in ecology in order to evaluate the variety and balance of multiword expression annota- tion produced by automatic annotation systems. The measures of richness, normalized richness, and two variations of Hill’s evenness are con- sidered in this paper. We observe how these measures behave against increasingly smaller samples of gold annotations of multiword ex- pressions and use their comportment to validate or invalidate their pertinence for multiword ex- pressions in annotated texts. We apply the vali- dated measures to annotations in 14 languages produced by systems during the PARSEME shared task on automatic identification of mul- tiword expressions and on the gold versions of the corpora. We also explore the limits of such evaluation by studying the impact of lemmati- zation errors in the Turkish corpus used in the shared task. 1 Introduction Diversity of naturally occurring phenomena and artefacts is a desirable property of many environ- ments and systems. It has been modelled and mea- sured in many domains, including linguistics but has rarely been formalized with respect to particu- lar linguistic phenomena within one language. This paper addresses diversity of one particular phe- nomenon: multiword expressions (MWEs), which are combinations of words, such as out of the blue or pay a visit, exhibiting idiosyncratic properties at lexical, morphological, syntactic and/or semantic level (Baldwin and Kim, 2010). Automatic annota- tion of multiword expression occurrences in texts – henceforth referred to as MWE identification fol- lowing nomenclature from Constant et al. (2017) – has been the focus of many works, among which the PARSEME shared tasks (Savary et al., 2017; Ramisch et al., 2018, 2020). During these tasks, performances of participating systems were evalu- ated on the precision, recall, and F1-score of their annotations. In order to get a better understanding of how par- ticipating MWE identifiers behave, performances were also measured for specific subtasks such as the annotation of light verb construction occurrences (pay a visit, give a lecture), or the annotation of MWEs that were not seen during training. Such analysis of the resulting annotation stems from a need to make sure that the systems getting the best scores are not simply performing well on a few eas- ier, more rewarding subtasks while ignoring others. Thus, these evaluation scenarios implicitly address some aspects of data and system diversity. In this paper we follow the same objectives but we address diversity explicitly and formally. We suggest that diversity measures could later be used to put performance measures in perspective, by favoring NLP tools which cover diverse types and not only easy and repetitive cases. The …

Content truncated to 500 words