A Neural Network Architecture for Multilingual Punctuation Generation

Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 1048–1053, Austin, Texas, November 1-5, 2016. c⃝2016 Association for Computational Linguistics A Neural Network Architecture for Multilingual Punctuation Generation Miguel Ballesteros1 Leo Wanner1,2 1NLP Group, Universitat Pompeu Fabra, Barcelona, Spain 2 Catalan Institute for Research and Advanced Studies (ICREA) miguel.ballesteros@upf.edu leo.wanner@upf.edu Abstract Even syntactically correct sentences are per- ceived as awkward if they do not contain cor- rect punctuation. Still, the problem of au- tomatic generation of punctuation marks has been largely neglected for a long time. We present a novel model that introduces punc- tuation marks into raw text material with transition-based algorithm using LSTMs. Un- like the state-of-the-art approaches, our model is language-independent and also neutral with respect to the intended use of the punctuation. Multilingual experiments show that it achieves high accuracy on the full range of punctuation marks across languages. 1 Introduction Although omnipresent in (language learner) gram- mar books, punctuation received much less atten- tion in linguistics and natural language processing (Krahn, 2014). In linguistics, punctuation is gener- ally acknowledged to possess different functions. Its traditionally most studied function is that to encode prosody of oral speech, i.e., the prosodic rhetori- cal function; see, e.g., (Kirchhoff and Primus, 2014) and the references therein. In particular the comma is assumed to possess a strong rhetorical function (Nunberg et al., 2002). Its other functions are the grammatical function, which leads it to form a sep- arate (along with semantics, syntax, and phonology) grammatical submodule (Nunberg, 1990), and the syntactic function (Quirk et al., 1972), which makes it reflect the syntactic structure of a sentence. The different functions of punctuation are also re- flected in different tasks in natural language process- ing (NLP): introduction of punctuation marks into a generated sentence that is to be read aloud, restora- tion of punctuation in speech transcripts, parsing un- der consideration of punctuation, or generation of punctuation in written discourse. Our work is cen- tered in the last task. We present a novel punctuation generation algorithm that is based on the transition- based algorithm with long short-term memories (LSTMs) by Dyer et al. (2015) and character-based continuous-space vector embeddings of words using bidirectional LSTMs (Ling et al., 2015b; Ballesteros et al., 2015). The algorithm takes as input raw ma- terial without punctuation and effectively introduces the full range of punctuation symbols. Although in- tended, first of all, for use in sentence generation, the algorithm is function- and language-neutral, which makes it different, compared to most of the state- of-the-art approaches, which use function- and/or language-specific features. 2 Related Work The most prominent punctuation-related NLP task has been so far introduction (or restoration) of punc- tuation in speech transcripts. Most often, classifier models are used that are trained on n-gram mod- els (Gravano et al., 2009), on n-gram models en- riched by syntactic and lexical features (Ueffing et al., 2013) and/or by acoustic features (Baron et al., 2002; Kol´aˇr and Lamel, 2012). Tilk and Alum¨ae (2015) use a lexical …

Content truncated to 500 words