New Benchmark Corpus and Models for Fine-grained Event Classification: To BERT or not to BERT?

Proceedings of the 28th International Conference on Computational Linguistics, pages 6663–6678 Barcelona, Spain (Online), December 8-13, 2020 6663 New Benchmark Corpus and Models for Fine-grained Event Classification: To BERT or not to BERT? Jakub Piskorski Institute for Computer Science Polish Academy of Sciences Warsaw, Poland jpiskorski@gmail.com Jacek Haneczok Erste Group IT Vienna, Austria jacek.haneczok@gmail.com Guillaume Jacquet Joint Research Centre European Commission Isrpa, Italy guillaume.jacquet@ec.europa.eu Abstract We introduce a new set of benchmark datasets derived from ACLED data for fine-grained event classification and compare the performance of various state-of-the-art machine learning models on these datasets, including SVM based on TF-IDF character n-grams and neural context-free embeddings (GLOVE and FASTTEXT) as well as deep learning-based BERT with its contextual embeddings. The best results in terms of micro (94.3-94.9%) and macro F1 (86.0-88.9%) were obtained using BERT transformer, with simpler TF-IDF character n-gram based SVM being an interesting alternative. Further, we discuss the pros and cons of the considered benchmark models in terms of their robustness and the dependence of the classification performance on the size of training data. 1 Introduction Since an ever-growing amount of information on events of any type is transmitted via web in the form of free texts (e.g. online news) one has witnessed in the last decades an emergence of research on devel- opment of methods and tools for automated detection and extraction of structured information on events from textual sources (King and Lowe, 2003; Yangarber et al., 2008; Atkinson et al., 2011; Piskorski et al., 2011; Leetaru and Schrodt, 2013; Ward et al., 2013; Pastor-Galindo et al., 2020). One particular step in the event extraction process is event classification, i.e., assigning to a text snippet including event trigger an event type using a domain specific taxonomy, which is the main focus of this paper. While vast amount of tasks and challenges on automated event extraction, including event classifi- cation, has been organised over the years, relatively little work has been reported on approaches for fine-grained event classification. Furthermore, the existing freely available datasets used for training and evaluation purposes are rather of tiny size, ranging usually up to 5-10K events. Due to the emergence of deep learning-based approaches for the entire range of NLP tasks, there is a particular need to have larger event classification corpora in order to gain better insights into the performance of such methods and their comparison with shallow learning approaches, e.g., in terms of training data sizes required to obtain ‘ac- ceptable’ performance, types of embeddings and model robustness vis-a-vis different data characteristics. The rise of deep learning-based approaches allowing for model pre-training in an unsupervised manner using only plain text and then utilizing transfer learning (via re-using the pre-trained model and only fine-tuning it in a supervised manner), poses additional questions with regard to the required data sizes This work is licensed under a Creative Commons Attribution 4.0 International Licence. Licence details: http:// creativecommons.org/licenses/by/4.0/. The views expressed in this article are those of the authors and not necessarily those of Erste …

Content truncated to 500 words