Neural Semi-Markov Conditional Random Fields for Robust Character-Based Part-of-Speech Tagging

Proceedings of NAACL-HLT 2019, pages 2736–2743 Minneapolis, Minnesota, June 2 - June 7, 2019. c⃝2019 Association for Computational Linguistics 2736 Neural Semi-Markov Conditional Random Fields for Robust Character-Based Part-of-Speech Tagging Apostolos Kemos2∗ Heike Adel1,3∗ 1 Center for Information and Language Processing (CIS), LMU Munich, Germany 2 Department of Computer Engineering and Informatics, University of Patras, Greece 3 Bosch Center for Artificial Intelligence (BCAI), Renningen, Germany kemos@ceid.upatras.gr heike.adel@de.bosch.com inquiries@cislmu.org Hinrich Sch¨utze1 Abstract Character-level models of tokens have been shown to be effective at dealing with within- token noise and out-of-vocabulary words. However, they often still rely on correct token boundaries. In this paper, we propose to elim- inate the need for tokenizers with an end-to- end character-level semi-Markov conditional random field. It uses neural networks for its character and segment representations. We demonstrate its effectiveness in multilingual settings and when token boundaries are noisy: It matches state-of-the-art part-of-speech tag- gers for various languages and significantly outperforms them on a noisy English version of a benchmark dataset. Our code and the noisy dataset are publicly available at http: //cistern.cis.lmu.de/semiCRF. 1 Introduction Recently, character-based neural networks (NNs) gained popularity for different tasks, ranging from text classification (Zhang et al., 2015) and language modeling (Kim et al., 2016) to ma- chine translation (Luong and Manning, 2016). Character-level models are attractive since they can effectively model morphological variants of words and build representations even for unknown words, suffering less from out-of-vocabulary problems (Pinter et al., 2017). However, most character-level models still rely on tokenization and use characters only for cre- ating more robust token representations (Santos and Zadrozny, 2014; Lample et al., 2016; Ma and Hovy, 2016; Plank et al., 2016). This leads to high performance on well-formatted text or text with misspellings (Yu et al., 2017; Sakaguchi et al., 2017) but ties the performance to the quality of the tokenizer. While humans are very robust to * Work was done at Center for Information and Lan- guage Processing, LMU Munich. noise caused by insertion of spaces (e.g., “car ni- val”) or deletion of spaces (“deeplearning”), this can cause severe underperformance of machine learning models. Similar challenges arise for lan- guages with difficult tokenization, such as Chinese or Vietnamese. For text with difficult or noisy to- kenization, more robust models are needed. In order to address this challenge, we propose a model that does not require any tokenization. It is based on semi-Markov conditional random fields (semi-CRFs) (Sarawagi and Cohen, 2005) which jointly learn to segment (tokenize) and label the input (e.g., characters). To represent the character segments, we compare different NN approaches. In our experiments, we address part-of-speech (POS) tagging. However, our model is gener- ally applicable to other sequence-tagging tasks as well since it does not require any task-specific hand-crafted features. Our model achieves state- of-the-art results on the Universal Dependencies dataset (Nivre et al., 2015). To demonstrate its effectiveness, we evaluate it not only on English but also on languages with inherently difficult to- kenization, namely Chinese, Japanese and Viet- namese. We …

Content truncated to 500 words