Efficient Automatic Punctuation Restoration Using Bidirectional Transformers with Robust Inference

Proceedings of the 17th International Conference on Spoken Language Translation (IWSLT), pages 272–279 July 9-10, 2020. c⃝2020 Association for Computational Linguistics https://doi.org/10.18653/v1/P17 272 Efficient Automatic Punctuation Restoration Using Bidirectional Transformers with Robust Inference Maury Courtland, Adam Faulkner, Gayle McElvain Capital One, Vision and Language Technologies {maury.courtland, adam.faulkner, gayle.mcelvain} @capitalone.com Abstract Though people rarely speak in complete sen- tences, punctuation confers many benefits to the readers of transcribed speech. Unfortu- nately, most ASR systems do not produce punctuated output. To address this, we pro- pose a solution for automatic punctuation that is both cost efficient and easy to train. Our solution benefits from the recent trend in fine-tuning transformer-based language mod- els. We also modify the typical framing of this task by predicting punctuation for sequences rather than individual tokens, which makes for more efficient training and inference. Finally, we find that aggregating predictions across multiple context windows improves accuracy even further. Our best model achieves a new state of the art on benchmark data (TED Talks) with a combined F1 of 83.9, representing a 48.7% relative improvement (15.3 absolute) over the previous state of the art. 1 Introduction Enabling computers to use speech as input has long been an aspirational goal in the field of human com- puter interaction. Recent advances have had dra- matic impact across multiple domains (e.g. reliev- ing medical professionals from having to transcribe medical dictation (Edwards et al., 2017), improv- ing real-time spoken language translation (Gu et al., 2017), and affording convenience through conver- sational interfaces like those in virtual personal assistants (McTear et al., 2016)). For use cases that require reading transcribed speech, however, it is often still a challenge to recover meaningful clause boundaries from disfluent, errorful utterances. Humans rely on punctuation for readability, per- haps because it lessens the burden of ambiguous phrasing. Studies have found that removing punctu- ation from manual transcriptions can be even more detrimental to understanding than a word error rate of 15% or 20% (T¨undik et al., 2018). Reading comprehension is also significantly slower with- out punctuation (Jones et al., 2003). For down- stream NLP models, the lack of clausal boundaries can significantly decrease accuracy (e.g. a 4.6% BLEU decrease in NMT; Vandeghinste et al. 2018). This likely reflects the discrepancy between well- segmented training corpora and ASR output. To solve the lack of punctuation in ASR out- put, we propose an automatic punctuation model, which leverages the recent trend in unsupervised pre-training (Devlin et al., 2019) and the parallel ar- chitecture of transformer networks (Vaswani et al., 2017). Unsupervised pre-training dramatically re- duces the amount of labeled data required for su- perior performance on this task. Additionally, the model’s departure from a recurrent architecture al- lows direct connections between all input tokens. This enables the network to more easily model long- distance dependencies (e.g. on one hand, ... on the other, ...) for improved punctuation performance. The departure from a recurrent architecture also allows computations to be performed in parallel for each layer with the speed …

Content truncated to 500 words