Generative Models For Indic Languages: Evaluating Content Generation Capabilities

Proceedings of Recent Advances in Natural Language Processing, pages 187–195 Varna, Sep 4–6, 2023 https://doi.org/10.26615/978-954-452-092-2_021 187 Generative Models For Indic Languages: Evaluating Content Generation Capabilities Savita Bhat TCS Research IIIT Hyderabad savita.bhat@tcs.com Vasudeva Varma IIIT Hyderabad vv@iiit.ac.in Niranjan Pedanekar Sony Research India niranjan.pedanekar @gmail.com Abstract Large language models (LLMs) and generative AI have emerged as the most important areas in the field of natural language processing (NLP). LLMs are considered to be a key component in several NLP tasks, such as summarization, question-answering, sentiment classification, and translation. Newer LLMs, such as Chat- GPT, BLOOMZ, and several such variants, are known to train on multilingual training data and hence are expected to process and gener- ate text in multiple languages. Considering the widespread use of LLMs, evaluating their effi- cacy in multilingual settings is imperative. In this work, we evaluate the newest generative models (ChatGPT, mT0, and BLOOMZ) in the context of Indic languages. Specifically, we consider natural language generation (NLG) ap- plications such as summarization and question- answering in monolingual and cross-lingual set- tings. We observe that current generative mod- els have limited capability for generating text in Indic languages in a zero-shot setting. In con- trast, generative models perform consistently better on manual quality-based evaluation in In- dic languages and English language generation. Considering limited generation performance, we argue that these LLMs are not intended to use in zero-shot fashion in downstream appli- cations. 1 Introduction Since the release of instruction-based ChatGPT, large language models (LLM) have taken the lan- guage generation research landscape by storm. Re- cent transformations in natural language process- ing (NLP) are largely enabled by pretrained LLMs such as T5 (Raffel et al., 2020), GPT3 (Brown et al., 2020), and LLaMa (Touvron et al., 2023) to name a few1. These models demonstrate impressive results 1Henceforth, we use generative models and LLMs inter- changeably. in various NLP tasks, including language genera- tion (NLG). Accordingly, the use of such off-the- shelf LLMs in solving downstream applications, such as conversational agents and creative copy- writing, is rising. Secondly, although the perfor- mance has reached a near-human level, most of these works focus on European languages. Specif- ically, the latest generative models, such as Chat- GPT and Bard, generate near-perfect content in English and other high-resource languages. How- ever, English is not the native language for most of the world’s population. One prime example of this is India, where people interact in one of their na- tive languages daily. Considering India is the most populated country in the world2, it is imperative to evaluate the latest progress in NLP (and NLG) and the potential of LLMs to be used as-is in the downstream tasks with a focus on Indic languages. So far, the LLMs have shown considerable prowess in tackling monolingual applications. But with increasing globalization and demand for in- formation, research in cross-lingual approaches is gaining attention. This upcoming field consists of methods to enable information access across mul- tiple languages. With this work, we provide an initial …

Content truncated to 500 words