Informal Persian Universal Dependency Treebank

Proceedings of the 13th Conference on Language Resources and Evaluation (LREC 2022), pages 7096–7105 Marseille, 20-25 June 2022 © European Language Resources Association (ELRA), licensed under CC-BY-NC-4.0 7096 Informal Persian Universal Dependency Treebank Roya Kabiri, Simin Karimi, Mihai Surdeanu University of Arizona Tucson AZ, USA {royakabiri, karimi, msurdeanu}@email.arizona.edu Abstract This paper presents the phonological, morphological, and syntactic distinctions between formal and informal Persian, showing that these two variants have fundamental differences that cannot be attributed solely to pronunciation discrepancies. Given that informal Persian exhibits particular characteristics, any computational model trained on formal Persian is unlikely to transfer well to informal Persian, necessitating the creation of dedicated treebanks for this variety. We thus detail the development of the open-source Informal Persian Universal Dependency Treebank, a new treebank annotated within the Universal Dependencies scheme. We then investigate the parsing of informal Persian by training two dependency parsers on existing formal treebanks and evaluating them on out-of-domain data, i.e. the development set of our informal treebank. Our results show that parsers experience a substantial performance drop when we move across the two domains, as they face more unknown tokens and structures and fail to generalize well. Furthermore, the dependency relations whose performance deteriorates the most represent the unique properties of the informal variant. The ultimate goal of this study that demonstrates a broader impact is to provide a stepping-stone to reveal the significance of informal variants of languages, which have been widely overlooked in natural language processing tools across languages. Keywords: dependency parsing, annotation, colloquial Persian 1. Introduction Most studies in natural language processing in general and dependency parsing in particular have mainly focused on the formal variants of languages and disregarded the informal ones (if any), with the notable exception of Twitter parsing and related tasks (Kong et al., 2014; Bharti et al., 2015). Persian1, a southwestern Iranian language, has two distinct variants: the literary formal form that has been traditionally used in writing as it was regarded more prestigious and prescriptively accurate, and the conversational informal form typically used in oral communications.2 The phonological, morphological, and syntactic distinctions between the two variants reveal that these varieties have fundamental differences that cannot be attributed solely to pronunciation discrepancies, and that they are more like two closely related dialects than simply different styles, which is the case in some languages such as English for which informal corpora have been developed. Several studies have suggested that contemporary Persian is diglossic (Boyle, 1952; Jeremiás, 1984), a term first coined by Ferguson (1959) to describe a speech community that naturally uses two varieties of a language for different purposes. With the advent of social media, the restrictions against the use of the conversational variant in writing have been challenged, and speakers tend to write in the spoken informal variety rather than the written formal variety, notwithstanding the strong opposition from the academy of Persian language and literature. Using conversational Persian in the Web creates a new challenge for the analysis of informal texts, as current grammars, …

Content truncated to 500 words