Language Identification

Over the last decade technological advances have been made which en- able us to envision real-world applications of speech technologies. It is possible to foresee applications where the spoken query …...

The goal of this study is to evaluate the potential for using large vocabulary continuous speech recognition as an engine for automatically classifying utterances according to the lan- guage being …...

This paper proposes a practical approach employing n-gram models and error-correction rules for Thai key prediction and Thai-English language identification. The paper also proposes rule-reduction algorithm applying mutual information to …...

We present a simple method for language inde- pendent and task independent text categoriza- tion learning, based on character-level n-gram language models. Our approach uses simple information theoretic principles and …...

Language identification has been an interesting and fascinating issue in natural language processing for decades, and there have been many researches on it. However, most of the researches are for …...

Code-switching is an interesting linguistic phenomenon commonly observed in highly bilingual communities. It consists of mixing languages in the same conversational event. This paper presents results on Part-of-Speech tagging Spanish-English …...

As the arm of NLP technologies extends beyond a small core of languages, tech- niques for working with instances of lan- guage data across hundreds to thousands of languages may …...

AbstractThere are many accurate methods for language identification of long text samples, but identification of very short strings still presents a challenge. This paper studies a language identification task, in …...

Language identification is the task of identify- ing the language a given document is written in. This paper describes a detailed examina- tion of what models perform best under dif- …...

AbstractThe intuition and basic hypothesis that this paper explores is that names are more characteristic of their language than common words are, and that a single name can have enough …...

In this paper, we revisit the problem of language identification with the focus on proper discrimination between closely related languages. Strong similarities between certain languages make it very hard to …...

Multilingual speakers switch between lan- guages in online and spoken communication. Analyses of large scale multilingual data re- quire automatic language identification at the word level. For our experiments with …...

AbstractLanguage identification is the task of automatically detecting the language(s) present in a document based on the content of the document. In this work, we address the problem of detecting …...

AbstractThis paper describes the advances in the multilingual text and speech database GlobalPhone, a multilingual database of high-quality read speech with corresponding transcriptions and pronunciation dictionaries in 20 languages. GlobalPhone …...

Code-mixing is frequently observed in user generated content on social media, especially from multilingual users. The linguistic complexity of such content is compounded by presence of spelling vari- ations, transliteration …...

Abstractn this paper, we present a set of computational methods to identify the likeliness of a word being borrowed, based on the signals from social media. In terms of Spearman’s …...

AbstractIn language identification, a common first step in natural language processing, we want to automatically determine the language of some input text. Monolingual language identification assumes that the given document …...

In a multilingual or sociolingual configura- tion Intra-sentential Code Switching (ICS) or Code Mixing (CM) is frequently observed nowadays. In the world most of the people know more than one …...

AbstractLanguage identification for code-switching (CS), the phenomenon of alternating between two or more languages in conversations, has traditionally been approached under the assumption of a single language per token. However, …...

AbstractThis paper describes the Alibaba Machine Translation Group submissions to the WMT 2020 Shared Task on Parallel Corpus Filtering and Alignment. In the filtering task, three main methods are applied …...

AbstractLinguistic Code-switching (CS) is still an understudied phenomenon in natural language processing. The NLP community has mostly focused on monolingual and multi-lingual scenarios, but little attention has been given to …...

AbstractCode-switching is the use of more than one language in the same conversation or utterance. Recently, multilingual contextual embedding models, trained on multiple monolingual corpora, have shown promising results on …...

AbstractLarge text corpora are increasingly important for a wide variety of Natural Language Processing (NLP) tasks, and automatic language identification (LangID) is a core technology needed to collect such datasets …...

AbstractRecent trends in NLP research have raised an interest in linguistic code-switching (CS); modern approaches have been proposed to solve a wide range of NLP tasks on multiple language pairs. …...

AbstractMultilingual contextual embeddings, such as multilingual BERT and XLM-RoBERTa, have proved useful for many multi-lingual tasks. Previous work probed the cross-linguality of the representations indirectly using zero-shot transfer learning on …...

AbstractAutomatic speech processing applications often have to deal with the problem of aggregating local descriptors (i.e., representations of input speech data corresponding to specific portions across the time dimension) and …...

AbstractThis paper introduces OpusTools, a package for downloading and processing parallel corpora included in the OPUS corpus collection. The package implements tools for accessing compressed data in their archived release …...

AbstractLanguage identification is a well-known task for natural language documents. In this paper we explore search query language identification which is usually the first task before any other query understanding. …...

AbstractSlot-filling, Translation, Intent classification, and Language identification, or STIL, is a newly-proposed task for multilingual Natural Language Understanding (NLU). By performing simultaneous slot filling and translation into a single output …...

AbstractWith the proliferation of social media platforms, anonymous discussions together with easy online access, reports on offensive content have caused serious concern to both authorities and research communities. Although there …...

AbstractAutomatic Language Identification (LI) or Dialect Identification (DI) of short texts of closely related languages or dialects, is one of the primary steps in many natural language processing pipelines. Language …...

AbstractCode-mixing (CM) is a frequently observed phenomenon that uses multiple languages in an utterance or sentence. There are no strict grammatical constraints observed in code-mixing, and it consists of non-standard …...

AbstractCode-Mixing (CM) is a common phenomenon in multilingual societies. CM plays a significant role in technology and medical fields where terminologies in the native language are not available or known. …...

AbstractData filtering for machine translation (MT) describes the task of selecting a subset of a given, possibly noisy corpus with the aim to maximize the performance of an MT system …...

AbstractLanguage identification is one of the fundamental tasks in natural language processing that is a prerequisite to data processing and numerous applications. Low-resourced languages with similar typologies are generally confused …...

AbstractThis paper provides language identification models for low- and under-resourced languages in the Pacific region with a focus on previously unavailable Austronesian languages. Accurate language identification is an important part …...