Do Pretrained Contextual Language Models Distinguish between Hebrew Homograph Analyses?

Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics, pages 849–864 May 2-6, 2023 ©2023 Association for Computational Linguistics Do Pretrained Contextual Language Models Distinguish between Hebrew Homograph Analyses? Avi Shmidman1,2,† Cheyn Shmuel Shmidman2,‡ Dan Bareket1,‡ Moshe Koppel1,2,‡ Reut Tsarfaty1,† 1Bar Ilan University 2DICTA / Jerusalem, Israel †{avi.shmidman,reut.tsarfaty}@biu.ac.il ‡{cheynshmuel,dbareket,moishk}@gmail.com Abstract Semitic morphologically-rich languages (MRLs) are characterized by extreme word ambiguity. Because most vowels are omitted in standard texts, many of the words are homo- graphs with multiple possible analyses, each with a different pronunciation and different morphosyntactic properties. This ambiguity goes beyond word-sense disambiguation (WSD), and may include token segmentation into multiple word units. Previous research on MRLs claimed that standardly trained pre-trained language models (PLMs) based on word-pieces may not sufficiently capture the internal structure of such tokens in order to distinguish between these analyses. Taking Hebrew as a case study, we investigate the extent to which Hebrew homographs can be disambiguated and analyzed using PLMs. We evaluate all existing models for contextualized Hebrew embeddings on a novel Hebrew homograph challenge sets that we deliver. Our empirical results demonstrate that contemporary Hebrew contextualized embeddings outperform non-contextualized embeddings; and that they are most effective for disambiguating segmentation and mor- phosyntactic features, less so regarding pure word-sense disambiguation. We show that these embeddings are more effective when the number of word-piece splits is limited, and they are more effective for 2-way and 3-way ambiguities than for 4-way ambiguity. We show that the embeddings are equally effective for homographs of both balanced and skewed distributions, whether calculated as masked or unmasked tokens. Finally, we show that these embeddings are as effective for homograph disambiguation with extensive supervised training as with a few-shot setup. 1 Introduction Semitic morphologically-rich languages (MRLs) such as Arabic, Hebrew, and Aramaic are char- acterized by extreme ambiguity at the word level (Wintner, 2014; Tsarfaty et al., 2020). In a standard text, many (and often most) of the words will be homographs with multiple possible analyses. The high ambiguity derives from several factors. First, prepositions, conjunctions, accusative pronouns, and possessive pronouns are often seamlessly af- fixed to words. Next, vowels are generally omitted in written texts. Finally, proper nouns are not differ- entiated from common nouns (no capital letters). The task of distinguishing between Hebrew ho- mograph analyses is related to the general task of Word Sense Disambiguation (WSD) (Agirre and Edmonds, 2006; Navigli, 2009), yet it is more chal- lenging. In the standard case of WSD, a single orthographic form is associated with a single word that can be analyzed in terms of two or more senses; also, the analyses are generally pronounced iden- tically, and often have the same morphosyntactic properties (e.g bank of a river vs. savings bank). In contrast, in Semitic languages, the need for dis- ambiguation often goes beyond a determination of sense. Hebrew word ambiguities can be divided into three primary categories (Table 1): 1. Segmen- tation ambiguities, in which a given orthographic form may (or may …

Content truncated to 500 words