Subword-Level Language Identification for Intra-Word Code-Switching

Proceedings of NAACL-HLT 2019, pages 2005–2011 Minneapolis, Minnesota, June 2 - June 7, 2019. c⃝2019 Association for Computational Linguistics 2005 Subword-Level Language Identification for Intra-Word Code-Switching Manuel Mager1, ¨Ozlem C¸ etino˘glu1 and Katharina Kann2 1Institute for Natural Language Processing, University of Stuttgart, Germany 2Center for Data Science, New York University, USA {manuel.mager, ozlem}@ims.uni-stuttgart.de, kann@nyu.edu Abstract Language identification for code-switching (CS), the phenomenon of alternating between two or more languages in conversations, has traditionally been approached under the as- sumption of a single language per token. How- ever, if at least one language is morpholog- ically rich, a large number of words can be composed of morphemes from more than one language (intra-word CS). In this paper, we extend the language identification task to the subword level, such that it includes splitting mixed words while tagging each part with a language ID. We further propose a model for this task, which is based on a segmental recur- rent neural network. In experiments on a new Spanish–Wixarika dataset and on an adapted German–Turkish dataset, our proposed model performs slightly better than or roughly on par with our best baseline, respectively. Consid- ering only mixed words, however, it strongly outperforms all baselines. 1 Introduction In settings where multilingual speakers share more than one language, mixing two or more languages within a single piece of text, for example a tweet, is getting increasingly common (Grosjean, 2010). This constitutes a challenge for natural language processing (NLP) systems, since they are com- monly designed to handle one language at a time. Code-switching (CS) can be found in multiple non-exclusive variants. For instance, sentences in different languages can be mixed within one text, or words from different languages can be combined into sentences. CS can also occur on the subword level, when speakers combine mor- phemes from different languages (intra-word CS). This last phenomenon can mostly be found if at least one of the languages is morphologically rich. An example for intra-word CS between the Ro- (a) ne’iwa pecansadoxi¯ WIX MIXED my.brother you-are.tired.PPFV (b) ne’iwa pe cansado xi¯ WIX WIX ES WIX my.brother you-are tired PPFV ‘My brother, you are tired.’ Figure 1: Intra-word CS between Spanish and Wixarika, (a) standard LID for CS, (b) our task. PPFV stands for past perfective. mance language Spanish and the Yuto-Aztecan language Wixarika1 is shown in Figure 1. CS language identification (LID) , i.e., predict- ing the language of each token in a text, has at- tracted a lot of attention in recent years (cf. Solorio et al. (2014); Molina et al. (2016)). However, intra-word mixing is mostly not handled explic- itly: words with morphemes from more than one language are simply tagged with a mixed label. While this works reasonably well for previously studied language pairs, overlooking intra-word CS leads to a major loss of information for highly pol- synthetic languages. A mixed word is unknown for NLP systems, yet a single word contains much more information, cf. Figure 1 (b). Furthermore, we find intra-word CS to be much …

Content truncated to 500 words