Semantic Role Labeling with Pretrained Language Models for Known and Unknown Predicates

Proceedings of Recent Advances in Natural Language Processing, pages 619–628, Varna, Bulgaria, Sep 2–4, 2019. https://doi.org/10.26615/978-954-452-056-4_073 619 Semantic Role Labeling with Pretrained Language Models for Known and Unknown Predicates Daniil Larionov FRC CSC RAS / Moscow, Russia RUDN University / Moscow, Russia dslarionov@isa.ru Artem Shelmanov Skoltech / Moscow, Russia FRC CSC RAS / Moscow, Russia a.shelmanov@skoltech.ru Elena Chistova FRC CSC RAS / Moscow, Russia RUDN University / Moscow, Russia chistova@isa.ru Ivan Smirnov FRC CSC RAS / Moscow, Russia RUDN University / Moscow, Russia ivs@isa.ru Abstract We build the first full pipeline for seman- tic role labelling of Russian texts. The pipeline implements predicate identifica- tion, argument extraction, argument clas- sification (labeling), and global scoring via integer linear programming. We train su- pervised neural network models for argu- ment classification using Russian seman- tically annotated corpus – FrameBank. However, we note that this resource pro- vides annotations only to a very limited set of predicates. We combat the prob- lem of annotation scarcity by introducing two models that rely on different sets of features: one for “known” predicates that are present in the training set and one for “unknown” predicates that are not. We show that the model for “unknown” pred- icates can alleviate the lack of annota- tion by using pretrained embeddings. We perform experiments with various types of embeddings including the ones gener- ated by deep pretrained language models: word2vec, FastText, ELMo, BERT, and show that embeddings generated by deep pretrained language models are superior to classical shallow embeddings for argu- ment classification of both “known” and “unknown” predicates. 1 Introduction Semantic role labeling (SRL) is one of techniques for shallow semantic parsing of natural language texts that produces predicate-argument structures of sentences. Predicates bear the central meaning of a situation expressed by a text. In most semantic theories, predicates are verbs, verbal nouns, and some other verb forms. Arguments are phrases that fill meaning slots of a situation expressed by a predicate and define its essential details. They answer such questions as “who?”, “did what?”, “to whom?”, “with what?”, “where?”, “when?”, etc. It is said that arguments play semantic roles in a situation as roles define meanings of slots. Role meanings and sizes of role inventories vary in different semantic theories and annotated cor- pora. Converting a text into such shallow seman- tic structures helps to abstract from syntactic and morphological representations of sentences and is considered to be an important technique for natu- ral language understanding. In (Jurafsky and Mar- tin, 2009), this is demonstrated with the following example for a predicate break. • John [AGENT] broke the window [THEME]. • John [AGENT] broke the window [THEME] with a rock [INSTRUMENT]. • The rock [INSTRUMENT] broke the win- dow [THEME]. • The window [THEME] broke. • The window [THEME] was broken by John [AGENT]. Note, that despite the surface syntactic rep- resentations of these sentences differ, the core predicate-argument structure retains and only ad- justs to available situation details. Semantic role labeling has been shown to be …

Content truncated to 500 words