Does Meta-learning Help mBERT for Few-shot Question Generation in a Cross-lingual Transfer Setting for Indic Languages?

Proceedings of the 29th International Conference on Computational Linguistics, pages 4251–4257 October 12–17, 2022. 4251 Does Meta-learning Help mBERT for Few-shot Question Generation in a Cross-lingual Transfer Setting for Indic Languages? Aniruddha Roy1, Rupak Kumar Thakur,2∗, Isha Sharma1, Ashim Gupta3, Amrith Krishna4, Sudeshna Sarkar1 and Pawan Goyal1 1IIT Kharagpur, 2Google India 3University of Utah, 4Uniphore {aniruddha.roy,ishasharma}@iitkgp.ac.in,rupakthakur@google.com sudeshna@cse.iitkgp.ac.in,pawang@cse.iitkgp.ac.in Abstract Few-shot Question Generation (QG) is an im- portant and challenging problem in the Natural Language Generation (NLG) domain. Multi- lingual BERT (mBERT) has been successfully used in various Natural Language Understand- ing (NLU) applications. However, the ques- tion of how to utilize mBERT for few-shot QG, possibly with cross-lingual transfer, remains. In this paper, we try to explore how mBERT performs in few-shot QG (cross-lingual trans- fer) and also whether applying meta-learning on mBERT further improves the results. In our setting, we consider mBERT as the base model and fine-tune it using a seq-to-seq lan- guage modeling framework in a cross-lingual setting. Further, we apply the model agnostic meta-learning approach to our base model. We evaluate our model for two low-resource In- dian languages, Bengali and Telugu, using the TyDi QA dataset. The proposed approach con- sistently improves the performance of the base model in few-shot settings and even works bet- ter than some heavily parameterized models in some settings. Human evaluation also con- firms the effectiveness of our approach. 1 Introduction QG can be defined as the task of generating an ap- propriate question based on the answer tokens and the context. The previous state-of-the-art QG mod- els are built using neural networks (Du et al., 2017; Zhou et al., 2017; Zhao et al., 2018; Nema et al., 2019), and are trained on high-resource languages with availability of vast amount of manually an- notated data for training. Collecting and anno- tating such vast data for training on low-resource languages can be challenging and costly. Cross- lingual transfer learning has shown its effective- ness in many NLP applications (Kumar et al., 2019; Chi et al., 2019; Asai et al., 2021; Xie et al., ∗The author contributed to the paper when he was a stu- dent of IIT Kharagpur 2018) for addressing data scarcity, because it al- lows us to transmit domain knowledge from a high resource annotated source language to domain of desired target language by fine-tuning with data from a target domain with low resource availabil- ity. mBERT (Devlin et al., 2018) has been success- fully used in various NLU tasks (Wu and Dredze, 2019; Hu et al., 2020). However, utilizing mBERT for generation tasks with cross-lingual transfer re- mains unexplored, specifically for QG. In this paper, we examine the application of mBERT for QG with cross-lingual transfer. Specifically, we ask: 1) Despite the successful usage of various multilingual auto-regressive lan- guage models (Xue et al., 2020; Liu et al., 2020; Maurya et al., 2021), can mBERT, an encoder- based model with fewer parameters than these auto-regressive models, be used for QG with cross- lingual transfer? 2) In few-shot cross-lingual trans- …

Content truncated to 500 words