Exploring Transformers for Ranking Portuguese Semantic Relations

Proceedings of the 13th Conference on Language Resources and Evaluation (LREC 2022), pages 2573–2582 Marseille, 20-25 June 2022 © European Language Resources Association (ELRA), licensed under CC-BY-NC-4.0 2573 Exploring Transformers for Ranking Portuguese Semantic Relations Hugo Gonc¸alo Oliveira CISUC, Department of Informatics Engineering University of Coimbra, Portugal, hroliv@dei.uc.pt Abstract We explored transformer-based language models for ranking instances of Portuguese lexico-semantic relations. Weights were based on the likelihood of natural language sequences that transmitted the relation instances, and expectations were that they would be useful for filtering out noisier instances. However, after analysing the weights, no strong conclusions were taken. They are not correlated with redundancy, but are lower for instances with longer and more specific arguments, which may nevertheless be a consequence of their sensitivity to the frequency of such arguments. They did also not reveal to be useful when computing word similarity with network embeddings. Despite the negative results, we see the reported experiments and insights as another contribution for better understanding transformer language models like BERT and GPT, and we make the weighted instances publicly available for further research. Keywords: semantic relations, lexical patterns, transformer models, BERT, GPT 1. Introduction Even though distributional semantics and deep learn- ing are the current trend in Natural Language Pro- cessing (NLP), research on the automatic acquisition of semantic relations from large corpora and semi- structured sources has a long history, which, among others, lead to the development of several systems for Open Information Extraction from the Web (Etzioni et al., 2008), as well as large knowledge bases like DBPedia (Auer et al., 2007), YAGO (Tanon et al., 2020), or BabelNet (Navigli and Ponzetto, 2012). In opposition to distributional and neural language mod- els, where words and sequences are represented by vec- tors of numbers, in the previous, relations are repre- sented by triples of the kind ‘arg 1 related-to arg2’, and are thus interpretable by humans. However, the au- tomatic acquisition of relations can be a noisy process, and it is not always straightforward to discriminate be- tween good extractions and those that are irrelevant or simply incorrect. To help with the latter, there has been work on computing the confidence of extractions, e.g., based on the occurrence of the relation arguments in a large collection of text (Cederberg and Widdows, 2003; Downey et al., 2005; Cimiano and Wenderoth, 2007). On the other hand, the adoption of models based on transformers (hereafter, TLMs), like GPT (Brown et al., 2020) and BERT (Devlin et al., 2019), lead to un- precedented advances in a broad range of NLP tasks. Since the latter encode much linguistic and world knowledge, some authors (Petroni et al., 2019; Haviv et al., 2021) show that, to some extent, they can be used as knowledge bases, e.g., when used to fill blanks in given text (see a recent review on the topic (AlKhamissi et al., 2022)). Furthermore, as it happens for traditional language models, TLMs can be used for computing the likelihood of given sequences of text. In principle, …

Content truncated to 500 words