Data-Driven Morphological Analysis and Disambiguation for Morphologically Rich Languages and Universal Dependencies

Proceedings of COLING 2016, the 26th International Conference on Computational Linguistics: Technical Papers, pages 337–348, Osaka, Japan, December 11-17 2016. Data-Driven Morphological Analysis and Disambiguation for Morphologically Rich Languages and Universal Dependencies Amir More IDC Herzliya habeanf@gmail.com Reut Tsarfaty Open University of Israel reutts@openu.ac.il Abstract Parsing texts into universal dependencies (UD) in realistic scenarios requires infrastructure for morphological analysis and disambiguation (MA&D) of typologically different languages as a first tier. MA&D is particularly challenging in morphologically rich languages (MRLs), where the ambiguous space-delimited tokens ought to be disambiguated with respect to their constituent morphemes. Here we present a novel, language-agnostic, framework for MA&D, based on a transition system with two variants, word-based and morpheme-based, and a dedicated transition to mitigate the biases of variable-length morpheme sequences. Our experiments on a Modern Hebrew case study outperform the state of the art, and we show that the morpheme-based MD consistently outperforms our word-based variant. We further illustrate the utility and multilingual coverage of our framework by morphologically analyzing and disambiguating the large set of languages in the UD treebanks. 1 Problem Statement A decade following the emergence of statistical parsers for English (Charniak, 1996; Bod, 1995), the CoNLL data sets presented a new challenge: the development of data-driven statistical parsers that can be trained to parse any language given an appropriately annotated treebank (Buchholz and Marsi, 2006; Nivre et al., 2007). These data sets facilitated the development of accurate, language-agnostic, depen- dency parsers (Nivre et al. (2006), McDonald (2006) etc.), but not without shortcomings: they require that input tokens be morphologically analyzed and disambiguated in advance. This latter assumption breaks down in realistic parsing scenarios where the morphological analysis of an input token may consist of multiple syntactic words to participate in the parse tree (Tsarfaty et al., 2010). The universal dependencies (UD) initiative aims to remedy this by presenting a harmonized set of treebanks, now 54 and counting, with a unified annotation scheme and multilayered annotation. Specifically, UD data distinguishes the input space-delimited tokens from the (morpho)syntactic words that participate in the parse tree (Nivre et al., 2016). Efforts towards parsing texts into universal dependencies in realistic scenarios thus require language- agnostic infrastructure for automatic morphological analysis and disambiguation (MA&D) of data from typologically different languages. MA&D is particularly challenging in morphologically rich languages (MRLs), where space-delimited input tokens may have multiple analyses, only one relevant in context. This morphological ambiguity of a token is typically represented as a lattice. The term Morphological Disambiguation (MD) refers to selecting a single path through the morphological analysis (MA) lattice. In Semitic languages, MD is particularly challenging (Adler, 2007; Bar-haim et al., 2008; Shacham and Wintner, 2007; Pasha et al., 2014; Habash and Rambow, 2005). To illustrate, Figure 1 shows the MA lattice of the Hebrew phrase ‘bclm hneim’1 (literally: in-shadow-of-them the-pleasant, meaning: in their pleasant shadow). Different paths in the lattice represent different disambiguation decisions. In This work is licenced under a Creative Commons Attribution 4.0 International License. License details: http:// creativecommons.org/licenses/by/4.0/ 1We …

Content truncated to 500 words