Multilingual Model Using Cross-Task Embedding Projection

Proceedings of the 23rd Conference on Computational Natural Language Learning, pages 22–32 Hong Kong, China, November 3-4, 2019. c⃝2019 Association for Computational Linguistics 22 Multilingual model using cross-task embedding projection Jin Sakuma The University of Tokyo jsakuma@tkl.iis.u-tokyo.ac.jp Naoki Yoshinaga Institute of Industrial Science, the University of Tokyo ynaga@iis.u-tokyo.ac.jp Abstract We present a method for applying a neural net- work trained on one (resource-rich) language for a given task to other (resource-poor) lan- guages. We accomplish this by inducing a mapping from pre-trained cross-lingual word embeddings to the embedding layer of the neu- ral network trained on the resource-rich lan- guage. To perform element-wise cross-task embedding projection, we invent locally linear mapping which assumes and preserves the lo- cal topology across the semantic spaces before and after the projection. Experimental results on topic classification task and sentiment anal- ysis task showed that the fully task-specific multilingual model obtained using our method outperformed the existing multilingual mod- els with embedding layers fixed to pre-trained cross-lingual word embeddings.1 1 Introduction Deep neural networks have improved the accu- racy of various natural language processing (NLP) tasks by performing representation learning with massive annotated datasets. However, the anno- tations in NLP depend on the target language as well as the task, and it is unrealistic to prepare such extensive annotated datasets for every pair of lan- guage and task. As a result, we can only obtain an accurate model for a few resource-rich languages such as English. To overcome this problem, researchers have at- tempted to make models trained with massive an- notated datasets in a resource-rich language (here- after, source language) applicable to a resource- poor language (target language) that have no an- notated datasets (Ruder et al., 2019) (§ 2). These methods utilize language-universal word represen- tations, namely cross-lingual word embeddings, to 1All the code is available at: https://github.com/ jyori112/task-spec Task-specific word embeddings (embedding layer of trained model) General cross-lingual word embeddings terrible bad good wonderful merveilleux (wonderful) eat food dog cat wonderful good merveilleux (wonderful) bad terrible mauvaise (bad) dog cat chat (cat) chen (dog) eat food nourriture (food) manger (eat) Figure 1: Locally linear mapping for sentiment analy- sis task. The relationship between “merveilleux (won- derful)” and its neighboring English words, “wonder- ful” and “good,” are preserved after projection. absorb the differences among languages in the vo- cabularies of neural network models; specifically, these multilingual models are trained with embed- ding layers fixed to pre-trained cross-lingual word embeddings. However, because those embedding layers are not optimized for the target task, the re- sulting model cannot exploit the true potential of representation learning, as demonstrated by Kim (2014) and our experimental results (§ 5.1). We propose methods of projecting pre-trained cross-lingual word embeddings to word embed- dings of a fully task-specific neural network all of whose parameters are optimized to the train- ing data in a source language, to realize fully task-specific multilingual model (§ 3). In ad- dition to naive linear projection, we present an element-wise projection method inspired by lo- …

Content truncated to 500 words