Unsupervised Deep Language and Dialect Identification for Short Texts

Proceedings of the 28th International Conference on Computational Linguistics, pages 1606–1617 Barcelona, Spain (Online), December 8-13, 2020 1606 Unsupervised Deep Language and Dialect Identification for Short Texts Koustava Goswami, Rajdeep Sarkar, Bharathi Raja Chakravarthi, Theodorus Fransen, and John P. McCrae Insight SFI Research Centre for Data Analytics, Data Science Institute, National University of Ireland Galway {koustava.goswami, rajdeep.sarkar, bharathi.raja, theodorus.fransen, john.mccrae}@insight-centre.org Abstract Automatic Language Identification (LI) or Dialect Identification (DI) of short texts of closely related languages or dialects, is one of the primary steps in many natural language processing pipelines. Language identification is considered a solved task in many cases; however, in the case of very closely related languages, or in an unsupervised scenario (where the languages are not known in advance), performance is still poor. In this paper, we propose the Unsupervised Deep Language and Dialect Identification (UDLDI) method, which can simultaneously learn sentence embeddings and cluster assignments from short texts. The UDLDI model understands the sentence constructions of languages by applying attention to character relations which helps to optimize the clustering of languages. We have performed our experiments on three short- text datasets for different language families, each consisting of closely related languages or dialects, with very minimal training sets. Our experimental evaluations on these datasets have shown significant improvement over state-of-the-art unsupervised methods and our model has outperformed state-of-the-art LI and DI systems in supervised settings. 1 Introduction Automatic Language Identification (LI) and Dialect Identification (DI) has become a very crucial part of natural language processing (NLP) pipelines and is part of tasks such as language modelling, categorization and analysis of code-mixed datasets. Many researchers have carried out experiments in these domains both in speech and texts starting from House and Neuburg (1977). Unsupervised LI is an under-explored area, but is highly useful as it can exploit the large amount of unlabelled data and, more importantly, can be employed when the languages to be identified are not known in advance. However, unsupervised LI for short texts is a very difficult task, for which performance is still comparatively poor. Zhang et al. (2016) have explored unsupervised language identification but this does not work well with closely related short texts. Zaidan and Callison-Burch (2014) have worked on the identification of Arabic dialects. Ciobanu et al. (2018) studied German dialect identification. These works are mostly supervised works. The task is even harder when it comes to unsupervised DI. These previous works have generated some questions: What is the best way to construct sentence embeddings and how to cluster them efficiently for short texts when there is no labelled training data? How to address the hard task of DI and LI for closely related languages in an unsupervised way without any manual intervention? In this paper, we address these problems by taking inspiration from iterative clustering (Xie et al., 2016) and self-attention-based sentence embeddings from Lin et al. (2017). Both of the papers have shown good results on unsupervised clustering and sentence embeddings for a single language dataset whereas our goal …

Content truncated to 500 words