Universal Word Segmentation: Implementation and Interpretation

Universal Word Segmentation: Implementation and Interpretation Yan Shao, Christian Hardmeier, Joakim Nivre Department of Linguistics and Philology, Uppsala University {yan.shao, christian.hardmeier, joakim.nivre}@lingfil.uu.se Abstract Word segmentation is a low-level NLP task that is non-trivial for a considerable number of languages. In this paper, we present a sequence tagging framework and apply it to word segmentation for a wide range of lan- guages with different writing systems and ty- pological characteristics. Additionally, we in- vestigate the correlations between various ty- pological factors and word segmentation ac- curacy. The experimental results indicate that segmentation accuracy is positively related to word boundary markers and negatively to the number of unique non-segmental terms. Based on the analysis, we design a small set of language-specific settings and extensively evaluate the segmentation system on the Uni- versal Dependencies datasets. Our model ob- tains state-of-the-art accuracies on all the UD languages. It performs substantially better on languages that are non-trivial to segment, such as Chinese, Japanese, Arabic and He- brew, when compared to previous work. 1 Introduction Word segmentation is the initial step for most higher level natural language processing tasks, such as part-of-speech tagging (POS), parsing and machine translation. It can be regarded as the problem of correctly identifying word forms from a character string. Word segmentation can be very challenging, es- pecially for languages without explicit word bound- ary delimiters, such as Chinese, Japanese and Viet- namese. Even for space-delimited languages like English or Russian, relying on white space alone generally does not result in adequate segmentation as at least punctuation should usually be separated from the attached words. For some languages, the space-delimited units in the surface form are too coarse-grained and therefore often further analysed, as in the cases of Arabic and Hebrew. Even though language-specific word segmentation systems are near-perfect for some languages, it is still useful to have a single system that performs reasonably with no or minimum language-specific adaptations. Word segmentation standards vary substantially with different definitions of the concept of a word. In this paper, we will follow the teminologies of Universal Dependencies (UD), where words are de- fined as basic syntactic units that do not always coincide with phonological or orthographic words. Some orthographic tokens, known in UD as mul- tiword tokens, therefore need to be broken into smaller units that cannot always be obtained by split- ting the input character sequence.1 To perform word segmentation in the UD frame- work, neither rule-based tokenisers that rely on white space nor the naive character-level sequence tagging model proposed previously (Xue, 2003) are ideal. In this paper, we present an enriched sequence labelling model for universal word segmentation. It is capable of segmenting languages in very diverse written forms. Furthermore, it simultaneously iden- tifies the multiword tokens defined by the UD frame- work that cannot be resolved simply by splitting 1Note that this notion of multiword token has nothing to do with the notion of multiword expression (MWE) as discussed, for example, in Sag et al. (2002). the input character sequence. …

Content truncated to 500 words