Question-Answering in a Low-resourced Language: Benchmark Dataset and Models for Tigrinya

Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics Volume 1: Long Papers, pages 11857–11870 July 9-14, 2023 ©2023 Association for Computational Linguistics Question-Answering in a Low-resourced Language: Benchmark Dataset and Models for Tigrinya Fitsum Gaim Wonsuk Yang Hancheol Park Jong C. Park∗ School of Computing Korea Advanced Institute of Science and Technology {fitsum.gaim,dirrick0511,hancheol.park,jongpark}@kaist.ac.kr Abstract Question-Answering (QA) has seen significant advances recently, achieving near human-level performance over some benchmarks. However, these advances focus on high-resourced lan- guages such as English, while the task remains unexplored for most other languages, mainly due to the lack of annotated datasets. This work presents a native QA dataset for an East African language, Tigrinya. The dataset contains 10.6K question-answer pairs spanning 572 paragraphs extracted from 290 news articles on various topics. The dataset construction method is dis- cussed, which is applicable to constructing sim- ilar resources for related languages. We present comprehensive experiments and analyses of several resource-efficient approaches to QA, in- cluding monolingual, cross-lingual, and multi- lingual setups, along with comparisons against machine-translated silver data. Our strong base- line models reach 76% in the F1 score, while the estimated human performance is 92%, in- dicating that the benchmark presents a good challenge for future work. We make the dataset, models, and leaderboard publicly available.1 1 Introduction Question Answering (QA) and Machine Reading Comprehension (MRC) have seen significant ad- vances in recent years, achieving human-level per- formance on large-scale benchmarks (Rajpurkar et al., 2016, 2018). The main factors driving the progress are the adaption of large pre-trained large language models and the proliferation of QA datasets (Rogers et al., 2022). However, most stud- ies focus on high-resourced languages, while the task remains unexplored for most of the World’s diverse languages. The primary challenge for non- English QA is the lack of native annotated datasets. In particular, there is little to no study done on scarcely resourced languages such as Tigrinya that ∗ Coressponding author 1 TiQuAD: https://github.com/fgaim/tiquad Article: ◊yM §M” [The Red Sea] Paragraph: ◊yM §M” KÔ ¿bµm bƒr§‘e° Mb” zÕw‘U §Mr³t ‘A†m¹ ≈†t b» §M”, Јm §M”, ◊yM §M” ¼m∫‘wn Ó‘e× §M” …Ï ~ Œ¤£l sΩ d≈ ¿bµm ƒb gÌ k�m² bbzJ z•bI bf‰y d≈ t•yÁdz√m ‚”t”n z°§h‡ ŒrË≈t z√· °ƒÅsÑn bzhbÇ ◊yM Mb” mff· ynÚ�‡ ~ [The Red Sea is one of the four seas in the world that are named after common colors: the Yellow Sea, the Black Sea, the Red Sea and the White Sea. The origin of its name is attributed to the red color given by the poisonous bacteria, especially the Trichodesmium Erythraeum, which breed in large numbers during the Summer season.] Question 1: ƒb ‘A†m knÔy bMb” zÕw‘U §Mr³t ƒ†Ã? [How many seas in the World are named after common colors?] Answer: ƒr§‘e° [four] Question 2: ƒb ‘A†m bMb” zÕw‘U §Mr³t ±e‘s? [List all the seas in the World that are named after common colors?] Answer: b» §M”, Јm §M”, ◊yM §M” ¼m∫‘wn Ó‘e× §M” [Yellow Sea, the Black Sea, the Red Sea and …

Content truncated to 500 words