Probing Linguistic Features of Sentence-Level Representations in Neural Relation Extraction

Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 1534–1545 July 5 - 10, 2020. c⃝2020 Association for Computational Linguistics 1534 Probing Linguistic Features of Sentence-Level Representations in Neural Relation Extraction Christoph Alt Aleksandra Gabryszak Leonhard Hennig German Research Center for Artificial Intelligence (DFKI) Speech and Language Technology Lab {christoph.alt, aleksandra.gabryszak, leonhard.hennig}@dfki.de Abstract Despite the recent progress, little is known about the features captured by state-of-the-art neural relation extraction (RE) models. Com- mon methods encode the source sentence, con- ditioned on the entity mentions, before classi- fying the relation. However, the complexity of the task makes it difficult to understand how encoder architecture and supporting linguistic knowledge affect the features learned by the encoder. We introduce 14 probing tasks tar- geting linguistic properties relevant to RE, and we use them to study representations learned by more than 40 different encoder architecture and linguistic feature combinations trained on two datasets, TACRED and SemEval 2010 Task 8. We find that the bias induced by the architecture and the inclusion of linguistic fea- tures are clearly expressed in the probing task performance. For example, adding contextu- alized word representations greatly increases performance on probing tasks with a focus on named entity and part-of-speech informa- tion, and yields better results in RE. In con- trast, entity masking improves RE, but consid- erably lowers performance on entity type re- lated probing tasks. 1 Introduction Relation extraction (RE) is concerned with extract- ing relationships between entities mentioned in text, where relations correspond to semantic cate- gories such as org:founded by, person:spouse, or org:subsidiaries (Figure 1). Neural models have shown impressive results on this task, achieving state-of-the-art performance on standard datasets like SemEval2010 Task 8 (dos Santos et al., 2015; Wang et al., 2016; Lee et al., 2019), TA- CRED (Zhang et al., 2018; Alt et al., 2019b; Peters et al., 2019; Joshi et al., 2019), and NYT (Lin et al., 2016; Vashishth et al., 2018; Alt et al., 2019a). The majority of models implement an encoder architec- [...] included Aerolineas’s domestic subsidiary, Austral. org:subsidiaries tail (obj) head (subj) Figure 1: Example relation from TACRED. The sen- tence contains the relation org:subsidiaries between the head and tail entities ‘Aerolineas’ and ‘Austral’. ture to learn a fixed size representation of the input, e.g. a sentence, which is passed to a classification layer to predict the target relation label. These good results suggest that the learned rep- resentations capture linguistic and semantic prop- erties of the input that are relevant to the down- stream RE task, an intuition that was previously discussed for a variety of other NLP tasks by Con- neau et al. (2018). However, it is often unknown which exact properties the various models have learned. Our aim is to pinpoint the information a given RE model is relying on, in order to improve model performance as well as to diagnose errors. A general approach to model introspection is the use of probing tasks. Probing tasks (Shi et al., 2016; Adi et al., …

Content truncated to 500 words