Multilingual Word Segmentation: Training Many Language-Specific Tokenizers Smoothly Thanks to the Universal Dependencies Corpus

Multilingual Word Segmentation: Training Many Language-Specific Tokenizers Smoothly Thanks to the Universal Dependencies Corpus Erwan Moreau1, Carl Vogel2 1 Adapt Centre, 2 Trinity Centre for Computing and Language Studies, 1,2 Trinity College Dublin Trinity College Dublin, Dublin 2 {moreaue, vogel}@scss.tcd.ie Abstract This paper describes how a tokenizer can be trained from any dataset in the Universal Dependencies 2.1 corpus (UD2) (Nivre et al., 2017). A software tool, which relies on Elephant (Evang et al., 2013) to perform the training, is also made available. Beyond providing the community with a large choice of language-specific tokenizers, we argue in this paper that: (1) tokenization should be considered as a supervised task; (2) language scalability requires a streamlined software engineering process across languages. Keywords: Universal Dependencies, Word Segmentation, Tokenization, Multilinguality, Interoperability 1. Introduction This paper describes how a tokenizer can be trained from any dataset in the Universal Dependencies 2.1 corpus (UD2) (Nivre et al., 2017). A software tool, which relies on Elephant (Evang et al., 2013) to perform the training, is also made available. Beyond providing the community with a large choice of language-specific tokenizers, this pa- per explores two ideas: • A new perspective on word segmentation. We argue that tokenization does not depend only on language and genre,1 but includes conventions and choices re- lated to ambiguous cases within the same language- genre pair. As a consequence, using some prede- fined tokenizer might cause the loss of some infor- mation, especially when combining multiple language resources. It follows that, in general, tokenization should be seen as a supervised task, where textual data is tokenized following a given “model”: by making this model explicit, one can enforce model consistency when connecting various pre-tokenized language re- sources together, hence avoiding errors which might otherwise go undetected.2 • Towards language scalability. Major progress has been achieved in multilingual language technology in the recent years. In NLP applications, scalability in terms of data size has been addressed for the most part, but scalability in terms of language diversity is still a significant challenge. The Universal Dependen- cies 2.1 corpus includes 102 annotated datasets and 59 distinct languages, thanks to the authors’ and con- tributors’ great effort (Nivre et al., 2017). Packaging 1E.g. tokenizing tweets versus tokenizing The New York Times. 2This idea can be compared to the case of POS tags, for in- stance: it is clear that mixing datasets annotated with different la- beling schemes can lead to errors. In the case of tokenization, the scheme consists in the guidelines that the annotators follow for choosing which units represent tokens (otherwise stated, where the annotators choose to put the boudaries between tokens). such a diversity of languages in a uniform format is a major step towards the ability to process multiple languages in an homogeneous way, which is the cor- nerstone of language-wise scalability. Bikel and Zi- touni (2012, xxi) mention that “Previously, to build ro- bust and accurate multilingual natural language pro- cessing (NLP) applications, a researcher or developer had to consult …

Content truncated to 500 words