Span-based Named Entity Recognition by Generating and Compressing Information

Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics, pages 1984–1996 May 2-6, 2023 ©2023 Association for Computational Linguistics Span-based Named Entity Recognition by Generating and Compressing Information Nhung T. H. Nguyen1, Makoto Miwa2,3 and Sophia Ananiadou1,3,4 1National Centre for Text Mining, Department of Computer Science, The University of Manchester, United Kingdom 2Toyota Technological Institute, Japan 3Artificial Intelligence Research Center (AIRC), National Institute of Advanced Industrial Science and Technology (AIST), Japan 4Alan Turing Institute, United Kingdom {nhung.nguyen, sophia.ananiadou}@manchester.ac.uk makoto-miwa@toyota-ti.ac.jp Abstract The information bottleneck (IB) principle has been proven effective in various NLP applica- tions. The existing work, however, only used either generative or information compression models to improve the performance of the tar- get task. In this paper, we propose to combine the two types of IB models into one system to enhance Named Entity Recognition (NER). For one type of IB model, we incorporate two unsupervised generative components, span re- construction and synonym generation, into a span-based NER system. The span reconstruc- tion ensures that the contextualised span rep- resentation keeps the span information, while the synonym generation makes synonyms have similar representations even in different con- texts. For the other type of IB model, we add a supervised IB layer that performs informa- tion compression into the system to preserve useful features for NER in the resulting span representations. Experiments on five differ- ent corpora indicate that jointly training both generative and information compression mod- els can enhance the performance of the base- line span-based NER system. Our source code is publicly available at https://github.com/ nguyennth/joint-ib-models. 1 Introduction Tishby et al. (1999) introduced the information bottleneck (IB) method to compress representa- tion while preserving meaningful information. The method has been incorporated in many state-of- the-art (SOTA) deep models such as Variational Autoencoder (VAE) (Kingma and Welling, 2014; Rezende et al., 2014), and Deep Variational In- formation Bottleneck (Deep VIB) (Alemi et al., 2017). Those deep models can be divided into su- pervised generative models (e.g., Deep VIB) and unsupervised ones (e.g., VAEs) (Voloshynovskiy et al., 2019). Both VAE and VIB have been applied to NLP ap- plications. For example, Effland and Collins (2019) and Chen et al. (2018) proposed to use VAE in se- quence labelling tasks such as POS tagging and NER. Meanwhile, Wang et al. (2022) used VIB to tackle OOV issues in NER. The two types of IB have also been employed in other tasks, such as dialogue response generation (Chen et al., 2022), parsing (Li and Eisner, 2019), paraphrase gener- ation for MT (Ormazabal et al., 2022), and text summarisation (West et al., 2019), to name a few. Such previous work only used one type of IB model (either VAE or VIB) in their system, and it is un- clear whether we can effectively combine the two types of IB models. The NER task has been typically approached using sequence models such as BiLSTM (Lample et al., 2016) and BERT (Devlin et al., 2019; Lee et al., 2019; …

Content truncated to 500 words