Toward Interactive Dictation

Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics Volume 1: Long Papers, pages 15319–15338 July 9-14, 2023 ©2023 Association for Computational Linguistics Toward Interactive Dictation Belinda Li♠∗ Jason Eisner3 Adam Pauls3 Sam Thomson3 ♠MIT CSAIL 3Microsoft Semantic Machines ♠bzl@mit.edu 3{jason.eisner,adam.pauls,samuel.thomson}@microsoft.com Abstract Voice dictation is an increasingly important text input modality. Existing systems that al- low both dictation and editing-by-voice restrict their command language to flat templates in- voked by trigger words. In this work, we study the feasibility of allowing users to interrupt their dictation with spoken editing commands in open-ended natural language. We introduce a new task and dataset, TERTiUS, to experi- ment with such systems. To support this flexi- bility in real-time, a system must incrementally segment and classify spans of speech as either dictation or command, and interpret the spans that are commands. We experiment with using large pre-trained language models to predict the edited text, or alternatively, to predict a small text-editing program. Experiments show a natural trade-off between model accuracy and latency: a smaller model achieves 28% single- command interpretation accuracy with 1.3 sec- onds of latency, while a larger model achieves 55% with 7 seconds of latency. 1 Introduction Speech can be preferable for text entry, especially on mobile devices or while the user’s hands are occupied, and for some users for whom typing is always slow or impossible. While fast and accu- rate automatic speech recognition (ASR) is now ubiquitous (Kumar et al., 2012; Xiong et al., 2016; Chiu et al., 2018; Radford et al., 2022), ASR itself only transcribes speech. In practice, users may also wish to edit transcribed text. The ASR output might be incorrect; the user might have misspoken; or they might change their mind about what to say or how to phrase it, perhaps after seeing or hearing their previous version. Azenkot and Lee (2013) found that users with visual impairment spent 80% of time editing text vs. 20% dictating it. ∗ Work performed during a research internship at Mi- crosoft Semantic Machines. Just wanted to ask about the event on Friday the 23rd. Is the event still on? Just wanted to ask about the event on the 23rd, on Friday the 23rd. Is the event still on? Change“the event” to “it” in the last sentence. Just wanted to ask about the event on the 23rd. Just wanted to ask about the event on Friday the 23rd. Just wanted to check in about the event on Friday the 23rd. Is it still on? (c) (d) (b) (d) (a) (c) (a) (b) (START) cursor Figure 1: A user writes an email using speech in- put, interleaving dictation (a,c) and commanding (b,d). Top shows the continuous user utterance, while bottom shows the document state at each point of the utterance. Dictations are transcribed verbatim, while commands are interpreted and executed. Our system supports open- ended commanding (i.e., b,d both invoke a replace operation but use vastly different phrasing). In this work, we study the task of interactive …

Content truncated to 500 words