Splitting compounds with ngrams

Proceedings of COLING 2016, the 26th International Conference on Computational Linguistics: Technical Papers, pages 630–640, Osaka, Japan, December 11-17 2016. Splitting compounds with ngrams Naomi Tachikawa Shapiro Department of Linguistics Stanford University Stanford, CA 94305, USA tsnaomi@stanford.edu Abstract Compound words with unmarked word boundaries are problematic for many tasks in NLP and computational linguistics, including information extraction, machine translation, and syllabifica- tion. This paper introduces a simple, proof-of-concept language modeling approach to automatic compound segmentation, demonstrated with Finnish. The approach utilizes an off-the-shelf mor- phological analyzer to split training words into their constituent morphemes. A language model is subsequently trained on ngrams composed of morphemes, morpheme boundaries, and word boundaries. Finally, linguistic constraints are used to weed out phonotactically ill-formed seg- mentations, thereby allowing the language model to select the best grammatical segmentation. This approach achieves an accuracy of ∼97%. 1 Introduction Compound segmentation—the automatic splitting of compounds into their constituent words—is essen- tial to many language processing tasks, including machine translation, information extraction, semantic parsing, spell checking, and syllabification. To that end, we propose a simple, supervised approach to compound segmentation that integrates existing morphological analysis software with language model- ing and optional linguistic constraints. Specifically, the proposed segmenter seeks to identify word boundaries in closed compounds, com- pounds in which word boundaries go undelimited by spaces, hyphens, or other markers (e.g., bookworm). These are distinct from open compounds, where word boundaries are clearly indicated (e.g., rain gutter) and thus less of an issue for applications that require their whereabouts. (Note that all compounds are considered complex, while words that are not compounds are considered simplex—e.g., book, rain). In contrast to previous approaches, which have primarily sought to identify the dictionary forms of compound constituents, this approach aims to identify the exact location of word boundaries in com- pounds. This sort of identification is highly relevant to the domain of computational phonology. For instance, the location of word boundaries is crucial for syllabification. Closed compounds pose a problem for automatic syllabification because word boundaries form a proper subset of syllable bound- aries; a syllable break will always fall on a word boundary (or, so common phonological theory tells us). Without insight into the location of these boundaries, a rule-based syllabifier might fail to recognize compound-internal word boundaries. For example, if bookworm is mistaken for a simplex word, English phonotactics would syllabify it as *boo.kworm. Instead, we expect the syllabification book.worm, where the syllable break falls on the unmarked word boundary between book and worm. Hence, we have developed an approach to compound segmentation that specifically identifies word boundaries in compounds, versus lemmatized constituents. In Section 2, we describe previous ap- proaches to compound segmentation in the areas of machine translation and information extraction. We then give a broad sketch of our approach in Section 3, introducing morpheme-based language modeling. Finally, in Section 4, we describe and evaluate an implementation of our approach on Finnish data. This This work is licensed under a Creative Commons Attribution 4.0 International License. License …

Content truncated to 500 words