Strongly Incremental Repair Detection

Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 78–89, October 25-29, 2014, Doha, Qatar. c⃝2014 Association for Computational Linguistics Strongly Incremental Repair Detection Julian Hough1,2 1Dialogue Systems Group Faculty of Linguistics and Literature Bielefeld University julian.hough@uni-bielefeld.de Matthew Purver2 2Cognitive Science Research Group School of Electronic Engineering and Computer Science Queen Mary University of London m.purver@qmul.ac.uk Abstract We present STIR (STrongly Incremen- tal Repair detection), a system that de- tects speech repairs and edit terms on transcripts incrementally with minimal la- tency. STIR uses information-theoretic measures from n-gram models as its prin- cipal decision features in a pipeline of classifiers detecting the different stages of repairs. Results on the Switchboard dis- fluency tagged corpus show utterance-final accuracy on a par with state-of-the-art in- cremental repair detection methods, but with better incremental accuracy, faster time-to-detection and less computational overhead. We evaluate its performance us- ing incremental metrics and propose new repair processing evaluation standards. 1 Introduction Self-repairs in spontaneous speech are annotated according to a well established three-phase struc- ture from (Shriberg, 1994) onwards, and as de- scribed in Meteer et al. (1995)’s Switchboard cor- pus annotation handbook: John [ likes � �� � reparandum + {uh} � �� � interregnum loves ] � �� � repair Mary (1) From a dialogue systems perspective, detecting re- pairs and assigning them the appropriate structure is vital for robust natural language understanding (NLU) in interactive systems. Downgrading the commitment of reparandum phases and assigning appropriate interregnum and repair phases permits computation of the user’s intended meaning. Furthermore, the recent focus on incremental dialogue systems (see e.g. (Rieser and Schlangen, 2011)) means that repair detection should oper- ate without unnecessary processing overhead, and function efficiently within an incremental frame- work. However, such left-to-right operability on its own is not sufficient: in line with the princi- ple of strong incremental interpretation (Milward, 1991), a repair detector should give the best re- sults possible as early as possible. With one ex- ception (Zwarts et al., 2010), there has been no focus on evaluating or improving the incremental performance of repair detection. In this paper we present STIR (Strongly In- cremental Repair detection), a system which ad- dresses the challenges of incremental accuracy, computational complexity and latency in self- repair detection, by making local decisions based on relatively simple measures of fluency and sim- ilarity. Section 2 reviews state-of-the-art methods; Section 3 summarizes the challenges and explains our general approach; Section 4 explains STIR in detail; Section 5 explains our experimental set-up and novel evaluation metrics; Section 6 presents and discusses our results and Section 7 concludes. 2 Previous work Qian and Liu (2013) achieve the state of the art in Switchboard corpus self-repair detection, with an F-score for detecting reparandum words of 0.841 using a three-step weighted Max-Margin Markov network approach. Similarly, Georgila (2009) uses Integer Linear Programming post-processing of a CRF to achieve F-scores over 0.8 for reparan- dum start and repair start detection. However nei- ther approach can operate …

Content truncated to 500 words