Is it worth it? Budget-related evaluation metrics for model selection

Is it worth it? Budget-related evaluation metrics for model selection Filip Klubiˇcka, Giancarlo D. Salton, John D. Kelleher School of Computing Dublin Institute of Technology {filip.klubicka, giancarlo.salton}@mydit.ie john.d.kelleher@dit.ie Abstract Projects that set out to create a linguistic resource often do so by using a machine learning model that pre-annotates or filters the content that goes through to a human annotator, before going into the final version of the resource. However, available budgets are often limited, and the amount of data that is available exceeds the amount of annotation that can be done. Thus, in order to optimize the benefit from the invested human work, we argue that the decision on which predictive model one should employ depends not only on generalized evaluation metrics, such as accuracy and F-score, but also on the gain metric. The rationale is that, the model with the highest F-score may not necessarily have the best separation and sequencing of predicted classes, thus leading to the investment of more time and/or money on annotating false positives, yielding zero improvement of the linguistic resource. We exemplify our point with a case study, using real data from a task of building a verb-noun idiom dictionary. We show that in our scenario, given the choice of three systems with varying F-scores, the system with the highest F-score does not yield the highest profits. In other words, we show that the cost-benefit trade off can be more favorable if a system with a lower F-score is employed. Keywords: model evaluation, gain, budget, linguistic resource creation, idiom identification, idiom dictionary, F-score 1. Introduction Building linguistic resources, such as corpora or dictionar- ies, can be very labor-intensive, requiring great amounts of work-hours and expert annotation. However, as pointed out by Ringger et al. (2008), fixed budgets constrain the amount of annotation that can go into the construction of linguistic resources. In many cases, the amount of avail- able data far exceeds the time and money that is available for annotation, so one can only afford to label a subset of the data. Furthermore, for some linguistic resources only a particular (and sometimes rare) subset of the available data is relevant. Indeed, when the target linguistic phenomenon is relatively rare, the amount of time wasted filtering non- relevant data can be considerable. For example, consider the scenario of creating an idiom dictionary: in this context, only idiomatic phrases should be included in the dictionary and any time or effort expended on filtering non-idiomatic phrases is, essentially, an unnecessary cost to the project. Given that annotators are paid either by the hour or, more often, by the number of annotations they produce (mea- sured in words, sentences, phrases etc.), researchers are strongly motivated to reduce the annotation time, and thereby a project’s annotation costs, without sacrificing the quality and coverage of the resources they are creating. To this end, the annotation process is often supplemented with predictive models that to an extent streamline, speed up and ultimately cheapen the cost of human intervention. In …

Content truncated to 500 words