Event Causality Recognition Exploiting Multiple Annotators’ Judgments and Background Knowledge

Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing, pages 5816–5822, Hong Kong, China, November 3–7, 2019. c⃝2019 Association for Computational Linguistics 5816 Event Causality Recognition Exploiting Multiple Annotators’ Judgments and Background Knowledge Kazuma Kadowaki§‡, Ryu Iida§¶, Kentaro Torisawa§¶, Jong-Hoon Oh§, and Julien Kloetzer§ §Data-driven Intelligent System Research Center (DIRECT), National Institute of Information and Communications Technology (NICT) ‡Advanced Technology Laboratory, The Japan Research Institute, Limited ¶Graduate School of Science and Technology, Nara Institute of Science and Technology {kadowaki,ryu.iida,torisawa,rovellia,julien}@nict.go.jp Abstract We propose new BERT-based methods for recognizing event causality such as “smoke cigarettes” → “die of lung cancer” written in web texts. In our methods, we grasp each annotator’s policy by training multiple classi- fiers, each of which predicts the labels given by a single annotator, and combine the re- sulting classifiers’ outputs to predict the fi- nal labels determined by majority vote. Fur- thermore, we investigate the effect of sup- plying background knowledge to our classi- fiers. Since BERT models are pre-trained with a large corpus, some sort of background knowledge for event causality may be learned during pre-training. Our experiments with a Japanese dataset suggest that this is actually the case: Performance improved when we pre- trained the BERT models with web texts con- taining a large number of event causalities in- stead of Wikipedia articles or randomly sam- pled web texts. However, this effect was lim- ited. Therefore, we further improved perfor- mance by simply adding texts related to an in- put causality candidate as background knowl- edge to the input of the BERT models. We be- lieve these findings indicate a promising future research direction. 1 Introduction Event causality, such as “smoke cigarettes” → “die of lung cancer,” is critical knowledge for NLP applications such as machine reading (Rajpurkar et al., 2016). For the task of recognizing event causality written in web texts, we propose new BERT-based methods that exploit independent la- bels in a gold dataset provided by multiple annota- tors. In the creation of the dataset we used (Hashi- moto et al., 2014), three annotators independently labeled the data and the final labels were deter- mined by majority vote. In the previous work, each annotator’s independent judgments were ig- nored, but in our proposed method, we exploit each annotator’s judgments in predicting the ma- jority vote labels. The dataset we used had a reasonable degree of inter-annotator agreement (Fleiss’ Kappa value was 0.67), but a discrepancy remained among the annotators. Despite this discrepancy, we assume that their judgments are more or less consistent and that we can improve performance by training multiple classifiers, each from the labels provided by an individual annotator to grasp her/his policy, and by combining the resulting outputs of these classifiers. Researchers have studied how to exploit the differences between the behaviors of annotators and crowd workers to improve the quality of gold datasets (Snow et al., 2008; Zaidan and Callison- Burch, 2011; Zhou et al., 2012; …

Content truncated to 500 words