Recurrent Neural Networks with Mixed Hierarchical Structures and EM Algorithm for Natural Language Processing

Proceedings of the 13th Conference on Language Resources and Evaluation (LREC 2022), pages 6104–6113 Marseille, 20-25 June 2022 © European Language Resources Association (ELRA), licensed under CC-BY-NC-4.0 6104 Recurrent Neural Networks with Mixed Hierarchical Structures and EM Algorithm for Natural Language Processing Zhaoxin Luo, Michael Zhu Purdue University {luo293, yuzhu}@purdue.edu Abstract How to obtain hierarchical representations with an increasing level of abstraction becomes one of the key issues of learning with deep neural networks. A variety of RNN models have recently been proposed to incorporate both explicit and implicit hierarchical information in modeling languages in the literature. In this paper, we propose a novel approach called the latent indicator layer to identify and learn implicit hierarchical information (e.g., phrases), and further develop an EM algorithm to handle the latent indicator layer in training. The latent indicator layer further simplifies a text’s hierarchical structure, which allows us to seamlessly integrate different levels of attention mechanisms into the structure. We called the resulting architecture as the EM-HRNN model. Furthermore, we develop two bootstrap strategies to effectively and efficiently train the EM-HRNN model on long text documents. Simulation studies and real data applications demonstrate that the EM-HRNN model with bootstrap training outperforms other RNN-based models in document classification tasks. The performance of the EM-HRNN model is comparable to a Transformer-based method called Bert-base, though the former is much smaller model and does not require pre-training. Keywords: Hierarchical structure, EM algorithm, Bootstrap, Document classification, Deep neural networks 1. Introduction Text classification is the process of assigning tags or categories to texts according to their contents and is one of the major tasks in Natural Language Process- ing (NLP) with broad applications such as sentiment analysis, topic labeling, and spam detection. An impor- tant intermediate step in text classification is text repre- sentation learning. Previous work uses various neural network models to learn text representation, including Convolution Neural Networks (CNNs) (Zhang et al., 2015), Recurrent Neural Networks (RNNs) (Schmid- huber, 1991), and attention mechanisms (Yang et al., 2016). Recently, how to obtain hierarchical representations with an increasing level of abstraction becomes one of the key issues of learning in deep neural networks. A variety of hierarchical RNNs have been proposed to in- corporate hierarchical representations in modeling lan- guages in the literature. (Yang et al., 2016) proposed to incorporate existing explicit text hierarchical informa- tion. In particular, they proposed to process documents at two levels, which are the word- and sentence-levels, respectively, and obtained promising results. Another approach to modeling hierarchical and tempo- ral representations is to use multiscale RNNs (Schmid- huber, 1992; El Hihi and Bengio, 1996; Koutnik et al., 2014). (Chung et al., 2016) proposed the Hier- archical Multiscale Recurrent Neural Networks (HM- RNNs) equipped with boundary detectors that can dis- cover underlying hierarchical structures without prior information. (Luo and Zhu, 2021) called hierarchical structures with and without prior information as static and dynamic hi- erarchical structures (or boundaries), respectively, and proposed to use RNNs with Mixed Hierarchical Struc- ture (MHS-RNN) to accommodate …

Content truncated to 500 words