Utilizing Subword Entities in Character-Level Sequence-to-Sequence Lemmatization Models

Proceedings of the 28th International Conference on Computational Linguistics, pages 4676–4682 Barcelona, Spain (Online), December 8-13, 2020 4676 Utilizing Subword Entities in Character-Level Sequence-to-Sequence Lemmatization Models Nasser Zalmout and Nizar Habash Computational Approaches to Modeling Language (CAMeL) Lab New York University Abu Dhabi United Arab Emirates {nasser.zalmout,nizar.habash}@nyu.edu Abstract In this paper we present a character-level sequence-to-sequence lemmatization model, utilizing several subword features in multiple configurations. In addition to generic n-gram embeddings (using FastText), we experiment with concatenative (stems) and templatic (roots and patterns) morphological subwords. We present several architectures that embed these features directly at the encoder side, or learn them jointly at the decoder side with a multitask learning architecture. The results indicate that using the generic n-gram embeddings (through FastText) outperform the other linguistically-driven subwords. We use Modern Standard Arabic and Egyptian Arabic as test cases, with up to 22% and 13% relative error reduction, respectively, from a strong baseline. An error analysis shows that our best system is even able to handle word/lemma pairs that are both unseen in the training data. 1 Introduction Lemmatization is an important natural language processing task that maps inflected and possibly ambigu- ous word forms to abstract lemmas. Lemmatization is particularly important for modeling morphologically rich languages to address the sparsity of inflected forms. Data-driven lemmatization models have the challenge of generalizability beyond the training data, to handle unseen words, unseen lemmas and new contexts. Semitic languages, like Arabic and Hebrew, add other challenges: while words are generally writ- ten without disambiguating diacritical marks, unambiguous lemmas are expected to be fully diacritized; and there are many clitics that may be interpreted as part of the stem leading to alternative lemma readings in different contexts (see Table 1). Semitic languages are also known for their extensive use of templatic morphology which can greatly help in the task of lemmatization, except that words can have multiple pattern (template) readings also. As such, we expect that modeling subword features is a key to the lemmatization task. In this paper we experiment with several kinds of subword entities, without relying on expensive external resources. We experiment with generic n-gram embeddings, captured through FastText (Bojanowski et al., 2017), which generates word embeddings through n-gram sequence embedding. We also introduce and utilize greedy morphological subword features, which can be determined consistently at runtime without using any morphological dictionaries (Greedy columns in Table 1). We also use Morfessor (Creutz and Lagus, 2005) to obtain morpheme-like subwords. We compare several architectures that condition on these subwords at the encoder, or learn them jointly in a multitask learning architecture. The results indicate that the generic FastText embeddings matched or outperformed the other linguistically-driven subwords. These subwords are nevertheless considered linguistically shallow, as opposed to models using deeper – and more expensive – linguistic resources, like morphological analyzers (Zalmout and Habash, 2019; Pasha et al., 2014). We use Modern Standard Arabic (MSA) and Egyptian Arabic (EGY) as test cases, where our models provide about 22% relative error reduction for MSA, and …

Content truncated to 500 words