On the inconsistency of separable losses for structured prediction

Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics, pages 1491–1498 May 2-6, 2023 ©2023 Association for Computational Linguistics On the inconsistency of separable losses for structured prediction Caio Corro Universite Paris-Saclay, CNRS, LISN, 91400, Orsay, France caio.corro@lisn.upsaclay.fr Abstract In this paper, we prove that separable nega- tive log-likelihood losses for structured predic- tion are not necessarily Bayes consistent, or, in other words, minimizing these losses may not result in a model that predicts the most proba- ble structure in the data distribution for a given input. This fact opens the question of whether these losses are well-adapted for structured pre- diction and, if so, why. 1 Introduction Modern natural language processing (NLP) heavily relies on machine learning (ML), where prediction models are learned by minimizing a loss function over the training data. As such, loss functions play a central role in the design of these systems and it is important to understand their statistical properties in order to guarantee that the corresponding train- ing objectives are well defined. Although this topic is well studied in the ML community (Lugosi and Vayatis, 2004; Lin, 2004; Zhang, 2004a,b; Bartlett et al., 2006; Gneiting and Raftery, 2007; Liu, 2007; Tewari and Bartlett, 2007; Reid and Williamson, 2010; Williamson et al., 2016; Duchi et al., 2018; Blondel et al., 2020; Nowak et al., 2022), inter alia, there has been less focus on the structured predic- tion setting apart from a few recent works (Blondel, 2019; Nowak et al., 2019, 2020). In this paper, we emphasize the fact that, de- spite achievements in terms of accuracy, statisti- cal behavior of loss functions used in practice for structured prediction in NLP are not always well understood. We illustrate this fact by proving that commonly used separable loss functions for named entity recognition (NER) and dependency parsing are not Bayes consistent, meaning that training a model with these loss functions will not necessar- ily result in the prediction of the most the probable output for a given input in the data distribution. 2 Bayes consistency We denote inputs and outputs as x ∈ X and y ∈ Y , respectively. We assume each y ∈ Y is a binary vector whose elements are indexed by a set C, i.e. y ∈ {0, 1}C, where C is problem dependent. For example, in the k multiclass classification case, we have C = [k], where we use the shorthand [k] = {1, 2, ..., k}, and Y is defined as the set of standard bases (one-hot vectors) of dimension k, meaning that |Y | = k. More generally, the vector y is an indicator of “selected” parts in C and, in the structured prediction case, several parts can be jointly selected. Note that it is usual to assume that the parts in C can depend on the input x. Without loss of generality, we omit this detail as we will study loss functions in the pointwise setting. A scoring model f ∈ F …

Content truncated to 500 words