A Novel Challenge Set for Hebrew Morphological Disambiguation and Diacritics Restoration

Findings of the Association for Computational Linguistics: EMNLP 2020, pages 3316–3326 November 16 - 20, 2020. c⃝2020 Association for Computational Linguistics 3316 1 A Novel Challenge Set for Hebrew Morphological Disambiguation and Diacritics Restoration Avi Shmidman1,2, Joshua Guedalia1,2, Shaltiel Shmidman1,2, Moshe Koppel1,2, Reut Tsarfaty1,3 1Bar Ilan University / Ramat Gan, Israel 2DICTA / Jerusalem, Israel 3Allen Institute for Artificial Intelligence {avi.shmidman,josh.guedalia,shaltiel.shmidman, moshe.koppel,reut.tsarfaty}@biu.ac.il Abstract One of the primary tasks of morphological parsers is the disambiguation of homographs. Particularly difficult are cases of unbalanced ambiguity, where one of the possible analy- ses is far more frequent than the others. In such cases, there may not exist sufficient exam- ples of the minority analyses in order to prop- erly evaluate performance, nor to train effec- tive classifiers. In this paper we address the is- sue of unbalanced morphological ambiguities in Hebrew. We offer a challenge set for He- brew homographs — the first of its kind — containing substantial attestation of each anal- ysis of 21 Hebrew homographs. We show that the current SOTA of Hebrew disambiguation performs poorly on cases of unbalanced ambi- guity. Leveraging our new dataset, we achieve a new state-of-the-art for all 21 words, im- proving the overall average F1 score from 0.67 to 0.95. Our resulting annotated datasets are made publicly available for further research. 1 Introduction It is a known phenomenon that the distribution of linguistic units, or words, in a language follows a Zipf law distribution (Zipf, 1949), wherein a rel- atively small number of words appear frequently, and a much larger number of items appear in a long tail of words, as rare events (Czarnowska et al., 2019). Significantly, this also applies to the dis- tribution of analyses of a given homograph. Take for instance the simple POS-tag ambiguity in En- glish between noun and verb (Elkahky et al., 2018). The word “fair” can be used as an adjective (“a fair price”) or as a noun (“she went to the fair”). Yet, the distribution of these two analyses is certainly not fair; the adjectival usage is far more frequent than the nominal usage (e.g., in Bird et al. (2008) the latter is six times more frequent than the former). We will call such cases “unbalanced homographs”. Cases of unbalanced homographs pose a formidable challenge for automated morpholog- ical parsers and segmenters. In tagged training corpora, the frequent option will naturally domi- nate the overwhelming majority of the occurrences. If the training corpus is not sufficiently large, then the sparsity of the minority analysis will prevent generalization by machine-learning models. By the same token, it can be difficult to evaluate the per- formance of tagging systems regarding unbalanced homographs, because the sparsity of the minority analysis prevents computation of adequate scoring. The empirical consequences of unbalanced ho- mographs are magnified in morphologically rich languages (MRLs), including many Semitic lan- guages, where distinct morphemes are often affixed to the word itself, resulting in additional ambiguity (Fabri et al., 2014; Habash et al., 2009). Further- more, in many …

Content truncated to 500 words