Recognizing Semantic Relations by Combining Transformers and Fully Connected Models

Proceedings of the 12th Conference on Language Resources and Evaluation (LREC 2020), pages 5838–5845 Marseille, 11–16 May 2020 c⃝ European Language Resources Association (ELRA), licensed under CC-BY-NC 5838 Recognizing Semantic Relations by Combining Transformers and Fully Connected Models Dmitri Roussinov, Serge Sharoff, Nadezhda Puchnina University of Strathclyde, University of Leeds, University of Tallinn dmitri.roussinov@strath.ac.uk, s.sharoff@leeds.ac.uk, np486061@tlu.ee Abstract Automatically recognizing an existing semantic relation (e.g. “is a”, “part of”, “property of”, “opposite of” etc.) between two words (phrases, concepts, etc.) is an important task affecting many NLP applications and has been subject of extensive experimentation and modeling. Current approaches to automatically telling if a relation exists between two given concepts X and Y can be grouped into two types: 1) those modeling word-paths connecting X and Y in text and 2) those modeling distributional properties of X and Y separately, not necessary in the proximity to each other. Here, we investigate how both types can be improved and combined. We suggest a distributional approach that is based on an attention-based transformer. We have also developed a novel word path model that combines useful properties of a convolutional network with a fully connected language model. While our transformer-based approach works better, both our models significantly outperform the state-of-the-art within their classes of approaches. We also demonstrate that combining the two approaches results in additional gains since they use somewhat different data sources. 1. Introduction During the last few years, Recurrent Neural Networks (RNNs) and Convolutional Neural Networks (CNNs) have resulted in major breakthroughs and are behind the current state-of-the-art algorithms in language processing, com- puter vision, and speech recognition (LeCun et al., 2015). Meanwhile, modeling higher level abstract knowledge still remains a challenging problem even for them. This in- cludes classification of semantic relations: given a pair of concepts (words or word sequences) identify the best se- mantic label to describe their relationship. The possible labels are “is a”, “part-of”, “property-of”, “made-of”, etc. This information is useful in many applications. For ex- ample, knowing that London is a city can help a Question Answering system answer the question What cities does the River Thames go through? Information retrieval benefits from query expansion with more specific words, e.g. trans- portation disasters → railroad disasters. For the task of database federation, an attribute in one database (e.g. with values France, Germany, and UK) often needs to be auto- matically matched with an attribute called country in an- other database. The connection between word meanings and their usage is prominent in the theories of human cog- nition (Mikoajczak-Matyja, 2015) and human language ac- quisition (Bybee and Beckner, 2015). While manually cu- rated dictionaries exist, they are often out-of-date, not cov- ering specialized domains, designed to be used by people, and exist for only a few well resourced languages (English, German, etc.). Therefore, here we are interested in methods for automated discovery (knowledge acquisition, taxonomy mining, etc.) . The automated approaches to detecting semantic relations between concepts (words or phrases) can be divided into two major …

Content truncated to 500 words