Hierarchical Meta-Embeddings for Code-Switching Named Entity Recognition

Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing, pages 3541–3547, Hong Kong, China, November 3–7, 2019. c⃝2019 Association for Computational Linguistics 3541 Hierarchical Meta-Embeddings for Code-Switching Named Entity Recognition Genta Indra Winata, Zhaojiang Lin, Jamin Shin, Zihan Liu, Pascale Fung Center for Artificial Intelligence Research (CAiRE) Department of Electronic and Computer Engineering The Hong Kong University of Science and Technology, Clear Water Bay, Hong Kong {giwinata,zlinao,jmshinaa,zliucr}@connect.ust.hk, pascale@ece.ust.hk Abstract In countries that speak multiple main lan- guages, mixing up different languages within a conversation is commonly called code- switching. Previous works addressing this challenge mainly focused on word-level as- pects such as word embeddings. How- ever, in many cases, languages share com- mon subwords, especially for closely related languages, but also for languages that are seemingly irrelevant. Therefore, we pro- pose Hierarchical Meta-Embeddings (HME) that learn to combine multiple monolingual word-level and subword-level embeddings to create language-agnostic lexical representa- tions. On the task of Named Entity Recogni- tion for English-Spanish code-switching data, our model achieves the state-of-the-art perfor- mance in the multilingual settings. We also show that, in cross-lingual settings, our model not only leverages closely related languages, but also learns from languages with different roots. Finally, we show that combining dif- ferent subunits are crucial for capturing code- switching entities. 1 Introduction Code-switching is a phenomenon that often hap- pens between multilingual speakers, in which they switch between their two languages in conver- sations, hence, it is practically useful to recog- nize this well in spoken language systems (Winata et al., 2018a). This occurs more often for entities such as organizations or products, which motivates us to focus on the specific problem of Named En- tity Recognition (NER) in code-switching scenar- ios. We show one of the examples as the follow- ing: • walking dead le quita el apetito a cualquiera • (translation) walking dead (a movie title) takes away the appetite of anyone For this task, previous works have mostly focused on applying pre-trained word embed- dings from each language in order to represent noisy mixed-language texts, and combine them with character-level representations (Trivedi et al., 2018; Wang et al., 2018; Winata et al., 2018b). However, despite the effectiveness of such word- level approaches, they neglect the importance of subword-level characteristics shared across differ- ent languages. Such information is often hard to capture with word embeddings or randomly ini- tialized character-level embeddings. Naturally, we can turn towards subword-level embeddings such as FastText (Grave et al., 2018) to help this task, which will evidently allow us to leverage the mor- phological structure shared across different lan- guages. Despite such expected usefulness, there has not been much attention focused around using subword-level features in this task. This is partly because of the non-trivial difficulty of combin- ing different language embeddings in the sub- word space, which arises from the distinct seg- mentation into subwords for different languages. This leads us to explore the literature of Meta- Embeddings …

Content truncated to 500 words