Expert Concept-Modeling Ground Truth Construction for Word Embeddings Evaluation in Concept-Focused Domains

Proceedings of the 28th International Conference on Computational Linguistics, pages 6690–6702 Barcelona, Spain (Online), December 8-13, 2020 6690 Expert Concept-Modeling Ground Truth Construction for Word Embeddings Evaluation in Concept-Focused Domains Arianna Betti♠ Martin Reynaert♠♦ Thijs Ossenkoppele♠ Yvette Oortwijn♠♣ Andrew Salway♠♥ Jelke Bloem♠ ♠ Institute for Logic, Language and Computation. University of Amsterdam ♦ Tilburg School of Humanities and Digital Sciences. Tilburg University ♣ Algorithms, Geometry & Applications. Eindhoven University of Technology ♥ School of Media, Arts and Humanities. University of Sussex {ariannabetti, thijs.ossenkoppele, yvette.oortwijn}@gmail.com, {m.w.c.reynaert, a.j.salway, j.bloem}@uva.nl Abstract We present a novel, domain expert-controlled, replicable procedure for the construction of concept-modeling ground truths with the aim of evaluating the application of word embeddings. In particular, our method is designed to evaluate the application of word and paragraph embeddings in concept-focused textual domains, where a generic ontology does not provide enough information. We illustrate the procedure, and validate it by describing the construction of an expert ground truth, QuiNE-GT. QuiNE-GT is built to answer research questions concerning the concept of naturalized epistemology in QUINE, a 2-million-token, single-author, 20th-century English philosophy corpus of outstanding quality, cleaned up and enriched for the purpose. To the best of our ken, expert concept- modeling ground truths are extremely rare in current literature, nor has the theoretical methodology behind their construction ever been explicitly conceptualised and properly systematised. Expert-controlled concept-modeling ground truths are however essential to allow proper evaluation of word embeddings techniques, and increase their trustworthiness in specialised domains in which the detection of concepts through their expression in texts is important. We highlight challenges, requirements, and prospects for future work. 1 Introduction A distinctive aspect of language is its capacity to express ideas and encode concepts, enabling us to treat certain natural language features as proxies for concepts and conceptual structures. Mining concepts in text corpora is a valuable enterprise in many domains, but it is no trivial feat, as the link between concepts and the words expressing them cannot be (fully) automatised. In generic domains, concept mining can sometimes rely on resources that fix word-concept links, such as WordNet (Fellbaum, 1998) and thesauri (Kipfer, 2019). In expert domains, however, harder-to-come-by, specialised resources are needed. The issue is exacerbated in domains focusing on concepts and small corpora with high lexical variation and fine-grained meaning distinctions, like the history of the sciences and the humanities, and more broadly the history of ideas. Importantly, the word-concept link problem is not limited to the humanities, or fields studying shifting meanings or changing concepts. It fundamentally applies to any corpora containing specialised terminology requiring substantial background knowledge for its decoding. Retrieval of relevant information in text depends on correctly identifying which terms express the concepts of interest. Any researcher or investigator struggling to sift through appreciably vast, lexically varied text collections to obtain specific, highly relevant information faces the problem, as do librarians and publishers wishing to give recommendations based on the ideas expressed in full- text documents, rather than based on metadata or user profile (Lops et …

Content truncated to 500 words