Discriminating Similar Languages: Evaluations and Explorations

Discriminating Similar Languages: Evaluations and Explorations Cyril Goutte1, Serge L´eger1, Shervin Malmasi2, Marcos Zampieri3,4 National Research Council (NRC), Canada1, Macquarie University, Australia2 Saarland University, Germany3, German Research Center for Artificial Intelligence (DFKI), Germany4 firstname.lastname@nrc.ca, shervin.malmasi@mq.edu.au, marcos.zampieri@uni-saarland.de Abstract We present an analysis of the performance of machine learning classifiers on discriminating between similar languages and language varieties. We carried out a number of experiments using the results of the two editions of the Discriminating between Similar Languages (DSL) shared task. We investigate the progress made between the two tasks, estimate an upper bound on possible performance using ensemble and oracle combination, and provide learning curves to help us understand which languages are more challenging. A number of difficult sentences are identified and investigated further with human annotation. Keywords: language identification, language varieties, evaluation 1. Introduction Discriminating between similar languages and language varieties is one of the main challenges of state-of-the-art language identification systems (Tiedemann and Ljubeˇsi´c, 2012). Closely-related languages such as Indonesian and Malay or Croatian and Serbian are very similar both at their spoken and at their written forms making it difficult for sys- tems to discriminate between them. Varieties of the same language, e.g. Spanish from South America or Spain, are even more difficult to detect than similar languages. Nev- ertheless, in both cases, recent work has shown that it is possible to train algorithms to discriminate between sim- ilar languages and language varieties with high accuracy (Goutte et al., 2014; Malmasi and Dras, 2015b). This study looks in more detail into the features that help algorithms discriminating between similar languages, tak- ing into account the results of two recent editions of the Discriminating between Similar Languages (DSL) shared task (Zampieri et al., 2014; Zampieri et al., 2015b). The analysis and results of this paper complement the informa- tion presented in the shared task reports and provide novel and important information for researchers and developers interested in language identification and particularly in the problem of discriminating between similar languages. 2. Related Work Language identification in written texts is a well- established research topic in computational linguistics. In- terest in the task is evidenced by early n-gram-based ap- proaches (Dunning, 1994; Grefenstette, 1995) to more re- cent studies (Brown, 2013; Lui et al., 2014a; Brown, 2014; Sim˜oes et al., 2014). The interest in the discrimination of similar languages, language varieties, and dialects is more recent but it has been growing in the past few years. Ex- amples of studies include the cases of Malay and Indone- sian (Ranaivo-Malanc¸on, 2006), Chinese varieties (Huang and Lee, 2008), South Slavic languages (Ljubeˇsi´c et al., 2007; Ljubeˇsi´c and Kranjˇci´c, 2015), Portuguese varieties (Zampieri and Gebre, 2012), Spanish varieties (Zampieri et al., 2013; Maier and G´omez-Rodrıguez, 2014), English varieties (Lui and Cook, 2013), Persian and Dari (Malmasi and Dras, 2015a), Romanian dialects (Ciobanu and Dinu, 2016), and a number of studies on Arabic dialects (Elfardy and Diab, 2014; Zaidan and Callison-Burch, 2014; Till- mann et al., 2014; Malmasi et al., 2015a) A number of shared …

Content truncated to 500 words