Fine-Tuning and Evaluation of Question Generation for Slovak Language

Fine-Tuning and Evaluation of Question Generation for Slovak Language Ondrej Megela Deutsche Telekom IT Solutions, Koˇsice, Slovakia ondrej.megela@telekom.com Daniel Hl´adek Technical University of Koˇsice, Slovakia daniel.hladek@tuke.sk Mat´uˇs Pleva Technical University of Koˇsice, Slovakia matus.pleva@tuke.sk J´an Staˇs Technical University of Koˇsice, Koˇsice, Slovakia jan.stas@tuke.sk Ming-Hsiang Su Soochow University, Taipei, Taiwan huntfox.su@gmail.com Yuan-Fu Liao National Yang Ming Chiao Tung University, HsinChu, Taiwan yfliao@nycu.edu.tw Abstract Automatic generation of questions about the given context is useful for the adaptation of question-answering systems or to support edu- cation. We trained and evaluated a model that generates a question in the Slovak language. We have designed an automatic metric where an additional question-answering model is used to evaluate the generated questions. We cal- culated how many questions have confidence greater than the given threshold. For generat- ing questions, we used contexts from the Slo- vak question-answering dataset. The fine-tuned Slovak T5 model did generate 38% of the ques- tions that the evaluation model could answer with confidence greater than 50%. We coop- erated with partners from Taiwan during these experiments in the frame of a bilateral project and we plan to transfer the knowledge to the Chinese language later. Keywords: evaluation, natural language genera- tion, neural networks, question answering, question generation 1 Introduction The idea of natural language processing (NLP) is to teach the computer to understand and respond to the user in natural language and thus prepare the user for comfortable communication. Natural language generation (NLG) refers to the process of automatically generating human-understandable text in one or more natural languages. The ability of a machine to generate text in natural language that is indistinguishable from that generated by humans is considered a prerequisite for artificial intelligence (AI). The onset of deep learning had a great impact on this area. Indeed, not only has it advanced the state- of-the-art in existing NLG tasks but has sparked interest in solving newer tasks. NLG today in- cludes a much wider range of tasks (Zhang et al., 2022) such as machine translation, text summa- rization, structured data-to-text generation, dialog generation, question answering, automatic ques- tion generation, video captioning, image descrip- tion, grammar correction, or automatic source code generation. The rapid progress of NLG in recent years can be attributed to 3 factors: 1. by developing data sets and benchmarks that allow training models (the more data the bet- ter); 2. advances in machine and deep learning algo- rithms have helped stabilize and accelerate large-model training; 3. availability of powerful and relatively cheaper computing infrastructure in the cloud space. The question of how to evaluate progress be- comes very important with such rapid development. Of course, the generated text can be evaluated based on grammatical correctness, however, ac- cording to which criteria to evaluate which of the generated texts is better if both are grammatically correct. More specifically, how can it be convincingly argued that the new NLG system is better than ex- isting state-of-the-art systems? We can let people evaluate and compare multiple outputs. The eval- uation scores …

Content truncated to 500 words