Looking Beyond Label Noise: Shifted Label Distribution Matters in Distantly Supervised Relation Extraction

Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing, pages 3841–3850, Hong Kong, China, November 3–7, 2019. c⃝2019 Association for Computational Linguistics 3841 Looking Beyond Label Noise: Shifted Label Distribution Matters in Distantly Supervised Relation Extraction Qinyuan Ye∗ University of Southern California qinyuany@usc.edu Liyuan Liu∗ University of Illinois, Urbana-Champaign ll2@illinois.edu Maosen Zhang Purdue University maosenzhang.milo@gmail.com Xiang Ren University of Southern California xiangren@usc.edu Abstract In recent years there is a surge of interest in applying distant supervision (DS) to auto- matically generate training data for relation extraction (RE). In this paper, we study the problem what limits the performance of DS-trained neural models, conduct thorough analyses, and identify a factor that can influence the performance greatly, shifted label distribution. Specifically, we found this problem commonly exists in real-world DS datasets, and without special handing, typical DS-RE models cannot automatically adapt to this shift, thus achieving deteriorated performance. To further validate our intuition, we develop a simple yet effective adaptation method for DS-trained models, bias adjust- ment, which updates models learned over the source domain (i.e., DS training set) with a label distribution estimated on the target domain (i.e., test set). Experiments demon- strate that bias adjustment achieves consistent performance gains on DS-trained models, es- pecially on neural models, with an up to 23% relative F1 improvement, which verifies our atassumptions. Our code and data can be found https://github.com/INK-USC/ shifted-label-distribution. 1 Introduction Aiming to identify the relation among an en- tity pair, relation extraction (RE) serves as an important step towards text understanding and is a long-standing pursuit by many researchers. To reduce the reliance on human-annotated data, especially for data-hungry neural models (Zeng et al., 2014; Zhang et al., 2017), there have been extensive studies on leveraging distant supervi- sion (DS) in conjunction with external knowledge bases to automatically generate large-scale train- ing data (Mintz et al., 2009; Zeng et al., 2015). ∗Equal contribution. children countries_of_residence country_of_birth country_of_death parents religion 0% 10% 20% p(ri| train) in KBP Train Test alternate_names city_of_headquarters country_of_headquarters dissolved founded founded_by 0% 1% 2% p(ri| train) in TACRED (6 classes) 0.00 0.25 0.50 KBP 0.00 0.25 0.50 NYT 0­0.01 0.01­0.03 0.03­0.05 0.05­0.1 0.1­0.2 >0.2 0.00 0.25 0.50 TACRED Figure 1: Left: Label distributions of KBP (distantly super- vised dataset) are shifted, while those of TACRED (human- annotated dataset, 6 classes for illustration) are consistent. Full distributions for NYT and TACRED can be found in Ap- pendix B. Right: Each relation ri is categorized into inter- vals along x-axis according to |p(ri|Dtrain) − p(ri|Dtest)|; height of the bars indicates proportion of test instances that fall into each category, revealing that the shift is severe in DS datasets such as KBP and NYT. While recent DS-based relation extraction meth- ods focus on handling label noise (Riedel et al., 2010; Hoffmann et al., 2011; Lin et al., 2016), i.e., false labels introduced by error-prone DS process, other factors may have been overlooked. Here, we observe model behaviors …

Content truncated to 500 words