A Generative Model for Punctuation in Dependency Trees

Transactions of the Association for Computational Linguistics, vol. 7, pp. 357–373, 2019. Action Editor: Trevor Cohn. Submission batch: 10/2018; Revision batch: 1/2019; Published 7/2019. c⃝ 2019 Association for Computational Linguistics. Distributed under a CC-BY 4.0 license. A Generative Model for Punctuation in Dependency Trees Xiang Lisa Li⇤and Dingquan Wang⇤ and Jason Eisner Department of Computer Science, Johns Hopkins University xli150@jhu.edu, {wdd,jason}@cs.jhu.edu Abstract Treebanks traditionally treat punctuation marks as ordinary words, but linguists have suggested that a tree’s “true” punctuation marks are not observed (Nunberg, 1990). These latent “underlying” marks serve to delimit or separate constituents in the syn- tax tree. When the tree’s yield is rendered as a written sentence, a string rewriting mech- anism transduces the underlying marks into “surface” marks, which are part of the ob- served (surface) string but should not be re- garded as part of the tree. We formalize this idea in a generative model of punc- tuation that admits efficient dynamic pro- gramming. We train it without observing the underlying marks, by locally maximiz- ing the incomplete data likelihood (simi- larly to the EM algorithm). When we use the trained model to reconstruct the tree’s underlying punctuation, the results appear plausible across 5 languages, and in par- ticular are consistent with Nunberg’s anal- ysis of English. We show that our gener- ative model can be used to beat baselines on punctuation restoration. Also, our recon- struction of a sentence’s underlying punctu- ation lets us appropriately render the surface punctuation (via our trained underlying-to- surface mechanism) when we syntactically transform the sentence. 1 Introduction Punctuation enriches the expressiveness of writ- ten language. When converting from spoken to written language, punctuation indicates pauses or pitches; expresses propositional attitude; and is conventionally associated with certain syntactic constructions such as apposition, parenthesis, quo- tation, and conjunction. In this paper, we present a latent-variable model of punctuation usage, inspired by the rule- based approach to English punctuation of Nun- berg (1990). Training our model on English data ⇤Equal contribution. learns rules that are consistent with Nunberg’s hand-crafted rules. Our system is automatic, so we use it to obtain rules for Arabic, Chinese, Spanish, and Hindi as well. Moreover, our rules are stochastic, which al- lows us to reason probabilistically about ambigu- ous or missing punctuation. Across the 5 lan- guages, our model predicts surface punctuation better than baselines, as measured both by per- plexity (§4) and by accuracy on a punctuation restoration task (§ 6.1). We also use our model to correct the punctuation of non-native writers of English (§ 6.2), and to maintain natural punc- tuation style when syntactically transforming En- glish sentences (§ 6.3). In principle, our model could also be used within a generative parser, al- lowing the parser to evaluate whether a candidate tree truly explains the punctuation observed in the input sentence (§8). Punctuation is interesting In The Linguistics of Punctuation, Nunberg (1990) argues that punctu- ation (in English) is more than a visual counter- part of spoken-language prosody, but forms a lin- guistic …

Content truncated to 500 words