Top-Rank-Focused Adaptive Vote Collection for the Evaluation of Domain-Specific Semantic Models

Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing, pages 3081–3093, November 16–20, 2020. c⃝2020 Association for Computational Linguistics 3081 Top-Rank-Focused Adaptive Vote Collection for the Evaluation of Domain-Specific Semantic Models Pierangelo Lombardo1, Alessio Boiardi1, Luca Colombo1, Angelo Schiavone1, and Nicol`o Tamagnone1 1 Intervieweb S.r.l. (Zucchetti group), Turin, Italy {pierangelo.lombardo, alessio.boiardi, luca.colombo, angelo.schiavone, nicolo.tamagnone}@intervieweb.it Abstract The growth of domain-specific applications of semantic models, boosted by the re- cent achievements of unsupervised embedding learning algorithms, demands domain-specific evaluation datasets. In many cases, content- based recommenders being a prime exam- ple, these models are required to rank words or texts according to their semantic related- ness to a given concept, with particular fo- cus on top ranks. In this work, we give a threefold contribution to address these require- ments: (i) we define a protocol for the con- struction, based on adaptive pairwise compar- isons, of a relatedness-based evaluation dataset tailored on the available resources and opti- mized to be particularly accurate in top-rank evaluation; (ii) we define appropriate metrics, extensions of well-known ranking correlation coefficients, to evaluate a semantic model via the aforementioned dataset by taking into ac- count the greater significance of top ranks. Fi- nally, (iii) we define a stochastic transitivity model to simulate semantic-driven pairwise comparisons, which confirms the effectiveness of the proposed dataset construction protocol. 1 Introduction In recent years, we have been witnessing a growth of Natural Language Processing (NLP) applica- tions in a wide range of specific domains, such as recruiting (INDA; Qin et al., 2018), law (Sugath- adasa et al., 2017), oil and gas (Nooralahzadeh et al., 2018), social media analysis (ALRashdi and O’Keefe, 2019), online education (Dess`ı et al., 2019), and biomedical (Patel et al., 2020). Embedding-based models have been playing a cru- cial role in this specialization, as they allow the ap- plication of the same learning algorithm to a variety of different corpora of unlabeled texts, obtaining domain-specific models (Bengio et al., 2003; Bo- janowski et al., 2017; Devlin et al., 2018; Mikolov et al., 2013a, 2017, 2013b; Pennington et al., 2014). The evaluation and validation of a domain- specialized model requires manually-annotated domain-specific datasets (Bakarov, 2018; Lastra- D´ıaz et al., 2019; Taieb et al., 2019). However, the construction of such datasets is a very resource- consuming process, and particular care is needed to ensure their ability to evaluate the desired fea- tures (Bakarov, 2018; Taieb et al., 2019; Wang et al., 2019). In particular, it is fundamental to carefully consider the so-called downstream task (i.e., the final purpose of the model), because the appropriate evaluation metric depends on this task (Bakarov, 2018; Blanco et al., 2013; Halpin et al., 2010; Rogers et al., 2018; Wang et al., 2019). Semantic similarity and relatedness are related but distinct notions in linguistics, the first being associated with concepts which share taxonomic properties and being maximized by synonyms; on the other hand, semantically related concepts can share any kind of semantic relation, including antonym (Cai et al., 2010; Harispe et al., …

Content truncated to 500 words