Multilingual Hierarchical Attention Networks for Document Classification

Proceedings of the The 8th International Joint Conference on Natural Language Processing, pages 1015–1025, Taipei, Taiwan, November 27 – December 1, 2017 c⃝2017 AFNLP Multilingual Hierarchical Attention Networks for Document Classification Nikolaos Pappas Idiap Research Institute Rue Marconi 19 CH-1920 Martigny, Switzerland nikolaos.pappas@idiap.ch Andrei Popescu-Belis HEIG-VD / HES-SO Route de Cheseaux 1 CH-1401 Yverdon, Switzerland andrei.popescu-belis@heig-vd.ch Abstract Hierarchical attention networks have re- cently achieved remarkable performance for document classification in a given lan- guage. However, when multilingual doc- ument collections are considered, train- ing such models separately for each lan- guage entails linear parameter growth and lack of cross-language transfer. Learn- ing a single multilingual model with fewer parameters is therefore a challenging but potentially beneficial objective. To this end, we propose multilingual hierarchical attention networks for learning document structures, with shared encoders and/or shared attention mechanisms across lan- guages, using multi-task learning and an aligned semantic space as input. We eval- uate the proposed models on multilingual document classification with disjoint la- bel sets, on a large dataset which we pro- vide, with 600k news documents in 8 lan- guages, and 5k labels. The multilingual models outperform monolingual ones in low-resource as well as full-resource set- tings, and use fewer parameters, thus con- firming their computational efficiency and the utility of cross-language transfer. 1 Introduction Learning word sequence representations has be- come increasingly useful for a variety of NLP tasks such as document classification (Tang et al., 2015; Yang et al., 2016), neural machine trans- lation (NMT) (Cho et al., 2014; Luong et al., 2015), question answering (Chen et al., 2015; Ku- mar et al., 2015) and summarization (Rush et al., 2015). However, when data are available in mul- tiple languages, representation learning must ad- Figure 1: Vectors of documents labeled with ‘Eu- rope’, ‘Culture’ and their Arabic counterparts. The multilingual hierarchical attention network separates topics better than monolingual ones. dress two main challenges. Firstly, the compu- tational cost of training separate models for each language, which grows linearly with their number, or even quadratically in the case of multi-way mul- tilingual NMT (Firat et al., 2016a). Secondly, the models should be capable of cross-language trans- fer, which is an important component in human language learning (Ringbom, 2007). For instance, Johnson et al. (2016) attempted to use a single sequence-to-sequence neural network model for NMT across multiple language pairs. Previous studies in document classification at- tempted to address these issues by employing multilingual word embeddings, which allow di- rect comparisons and groupings across languages (Klementiev et al., 2012; Hermann and Blunsom, 2014; Ferreira et al., 2016). However, they are only applicable when common label sets are avail- able across languages which is often not the case (e.g. Wikipedia or news). Moreover, despite re- cent advances in monolingual document modeling (Tang et al., 2015; Yang et al., 2016), multilingual models are still based on shallow approaches. In this paper, we propose Multilingual Hier- archical Attention Networks to learn shared doc- 1015 ument structures across languages for document classification with disjoint …

Content truncated to 500 words