Learning to Capitalize with Character-Level Recurrent Neural Networks: An Empirical Study

Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 2090–2095, Austin, Texas, November 1-5, 2016. c⃝2016 Association for Computational Linguistics Learning to Capitalize with Character-Level Recurrent Neural Networks: An Empirical Study Raymond Hendy Susanto† and Hai Leong Chieu‡ and Wei Lu† †Singapore University of Technology and Design ‡DSO National Laboratories {raymond susanto,luwei}@sutd.edu.sg chaileon@dso.org.sg Abstract In this paper, we investigate case restoration for text without case information. Previous such work operates at the word level. We pro- pose an approach using character-level recur- rent neural networks (RNN), which performs competitively compared to language model- ing and conditional random fields (CRF) ap- proaches. We further provide quantitative and qualitative analysis on how RNN helps im- prove truecasing. 1 Introduction Natural language texts (e.g., automatic speech tran- scripts or social media data) often come in non- standard forms, and normalization would typically improve the performance of downstream natural lan- guage processing (NLP) applications. This paper in- vestigates a particular sub-task in text normalization: case restoration or truecasing. Truecasing refers to the task of restoring case information (uppercase or lowercase) of characters in a text corpus. Case infor- mation is important for certain NLP tasks. For ex- ample, Chieu and Ng (2002) used unlabeled mixed case text to improve named entity recognition (NER) on uppercase text. The task often presents ambiguity: consider the word “apple” in the sentences “he bought an apple” and “he works at apple”. While the former refers to a fruit (hence, it should be in lowercase), the latter refers to a company name (hence, it should be cap- italized). Moreover, we often need to recover the case information for words that are previously un- seen by the system. In this paper, we propose the use of character- level recurrent neural networks for truecasing. Pre- vious approaches for truecasing are based on word level approaches which assign to each word one of the following labels: all lowercase, all upper- case, initial capital, and mixed case. For mixed case words, an additional effort has to be made to decipher exactly how the case is mixed (e.g., MacKenzie). In our approach, we propose a gen- erative, character-based recurrent neural network (RNN) model, allowing us to predict exactly how cases are mixed in such words. Our main contributions are: (i) we show that character-level approaches are viable compared to word-level approaches, (ii) we show that character- level RNN has a competitive performance compared to character-level CRF, and (iii) we provide our quantitative and qualitative analysis on how RNN helps improve truecasing. 2 Related Work Word-based truecasing The most widely used approach works at the word level. The simplest ap- proach converts each word to its most frequently seen form in the training data. One popular ap- proach uses HMM-based tagging with an N-gram language model, such as in (Lita et al., 2003; Nebhi et al., 2015). Others used a discriminative tagger, such as MEMM (Chelba and Acero, 2006) or CRF (Wang et al., 2006). Another approach uses statisti- cal machine …

Content truncated to 500 words