Direct Simultaneous Speech-to-Text Translation Assisted by Synchronized Streaming ASR

33778
Back to Articles Dashboard

Authors

nan

Abstract

Simultaneous speech-to-text translation is widely useful in many scenarios. The con- ventional cascaded approach uses a pipeline of streaming ASR followed by simultaneous MT, but suffers from error propagation and extra la- tency. To alleviate these issues, recent efforts attempt to directly translate the source speech into target text simultaneously, but this is much harder due to the combination of two separate tasks. We instead propose a new paradigm with the advantages of both cascaded and end- to-end approaches. The key idea is to use two separate, but synchronized, decoders on streaming ASR and direct speech-to-text trans- lation (ST), respectively, and the intermediate results of ASR guide the decoding policy of (but is not fed as input to) ST. During training time, we use multitask learning to jointly learn these two tasks with a shared encoder. En-to- De and En-to-Es experiments on the MuST- C dataset demonstrate that our proposed tech- nique achieves substantially better translation quality at similar levels of latency. 1

Keywords

nan

Full Content

Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021, pages 4618–4624 August 1–6, 2021. ©2021 Association for Computational Linguistics 4618 Direct Simultaneous Speech-to-Text Translation Assisted by Synchronized Streaming ASR ∗ Junkun Chen 1 Mingbo Ma 2 Renjie Zheng 2 Liang Huang 1,2 1Oregon State University, Corvallis, OR, USA 2Baidu Research, Sunnyvale, CA, USA chenjun2@oregonstate.edu, mingboma@baidu.com Abstract Simultaneous speech-to-text translation is widely useful in many scenarios. The con- ventional cascaded approach uses a pipeline of streaming ASR followed by simultaneous MT, but suffers from error propagation and extra la- tency. To alleviate these issues, recent efforts attempt to directly translate the source speech into target text simultaneously, but this is much harder due to the combination of two separate tasks. We instead propose a new paradigm with the advantages of both cascaded and end- to-end approaches. The key idea is to use two separate, but synchronized, decoders on streaming ASR and direct speech-to-text trans- lation (ST), respectively, and the intermediate results of ASR guide the decoding policy of (but is not fed as input to) ST. During training time, we use multitask learning to jointly learn these two tasks with a shared encoder. En-to- De and En-to-Es experiments on the MuST- C dataset demonstrate that our proposed tech- nique achieves substantially better translation quality at similar levels of latency. 1 Introduction Simultaneous speech-to-text translation incremen- tally translates source-language speech into target- language text, and is widely useful in many cross- lingual communication scenarios such as interna- tional travels and multinational conferences. The conventional approach to this problem is a cas- caded one (Arivazhagan et al., 2020; Xiong et al., 2019; Zheng et al., 2020b), involving a pipeline of two steps. First, the streaming automatic speech recognition (ASR) module transcribes the input speech on the fly (Moritz et al., 2020; Wang et al., 2020), and then a simultaneous text-to-text transla- tion module translates the partial transcription into target-language text (Oda et al., 2014; Dalvi et al., ∗ See our translation examples and demos at https://littlechencc.github.io/SimulST-demo/simulST-demo.html. Streaming ASR wait-2 unstable results wait-k policy streaming speech input extra delay wait-2 wait-k policy streaming speech input streaming speech input wait-k policy … Streaming ASR (a) (b) (c) wait-2 our method Ren et al,. 2020 Ma et al,. 2020 (a,b) cascaded <latexit sha1_base64="KvmtD2Qzol9BIWTg7UGnEKQGWco=">AB6nicbVBNS8NAEJ3Ur1q/qh69LBbBU0lE1GPRi8eK9gPaUDbTbt0swm7E6G/gQvH hTx6i/y5r9x2+agrQ8GHu/NMDMvSKQw6LrfTmFldW19o7hZ2tre2d0r7x80TZxqxhslrFuB9RwKRvoEDJ24nmNAokbwWjm6nfeuTaiFg94DjhfkQHSoSCUbTS/VP65UrbtWdgSwTLycVyFHvlb+6/ZilEVfIJDWm47kJ+hnVKJjk1I3NTyhbEQHvGOpohE3fjY7dUJOrNInYaxtKSQz9fdERiNjxlFgOyOKQ7PoTcX/vE6 K4ZWfCZWkyBWbLwpTSTAm079JX2jOUI4toUwLeythQ6opQ5tOyYbgLb68TJpnVe+i6t6dV2rXeRxFOIJjOAUPLqEGt1CHBjAYwDO8wpsjnRfn3fmYtxacfOYQ/sD5/AEQYo2n</latexit>z1 <latexit sha1_base64="NzBeE9vq5mC9DhZG+vA9j3WAOK8=">AB6nicbVBNS8NAEJ34WetX1aOXxSJ4KkR9Vj04rGi/YA2lM120y7dbMLuRKihP8GLB 0W8+ou8+W/ctjlo64OBx3szMwLEikMu63s7K6tr6xWdgqbu/s7u2XDg6bJk414w0Wy1i3A2q4FIo3UKDk7URzGgWSt4LRzdRvPXJtRKwecJxwP6IDJULBKFrp/qlX7ZXKbsWdgSwTLydlyFHvlb6/ZilEVfIJDWm47kJ+hnVKJjk2I3NTyhbEQHvGOpohE3fjY7dUJOrdInYaxtKSQz9fdERiNjxlFgOyOKQ7PoTcX/vE6 K4ZWfCZWkyBWbLwpTSTAm079JX2jOUI4toUwLeythQ6opQ5tO0YbgLb68TJrVindRce/Oy7XrPI4CHMJnIEHl1CDW6hDAxgM4Ble4c2Rzovz7nzMW1ecfOYI/sD5/AER5o2o</latexit>z2 <latexit sha1_base64="EFoKUrMUVpSXFxYRkZwK543aNE=">AB6nicbVBNS8NAEJ3Ur1q/qh69LBbBU0lU1GPRi8eK9gPaUDbSbt0swm7G6G/gQvH hTx6i/y5r9x2+ag1QcDj/dmJkXJIJr47pfTmFpeWV1rbhe2tjc2t4p7+41dZwqhg0Wi1i1A6pRcIkNw43AdqKQRoHAVjC6nvqtB1Sax/LejBP0IzqQPOSMGivdPfZOe+WKW3VnIH+Jl5MK5Kj3yp/dfszSCKVhgmrd8dzE+BlVhjOBk1I31ZhQNqID7FgqaYTaz2anTsiRVfokjJUtachM/TmR0UjrcRTYzoiaoV70puJ/Xic 14aWfcZmkBiWbLwpTQUxMpn+TPlfIjBhbQpni9lbChlRZmw6JRuCt/jyX9I8qXrnVf2rFK7yuMowgEcwjF4cAE1uIE6NIDBAJ7gBV4d4Tw7b87vLXg5DP78AvOxzcTao2p</latexit>z3 <latexit sha1_base64="gb5lGORDGwF6MLqPq9ofeWi6bHo=">AB6nicbVBNS8NAEJ3Ur1q/qh69LBbBU0lE1GPRi8eK9gPaUDbTbt0swm7E6G/gQvH hTx6i/y5r9x2+agrQ8GHu/NMDMvSKQw6LrfTmFldW19o7hZ2tre2d0r7x80TZxqxhslrFuB9RwKRvoEDJ24nmNAokbwWjm6nfeuTaiFg94DjhfkQHSoSCUbTS/VPvFeuFV3BrJMvJxUIEe9V/7q9mOWRlwhk9SYjucm6GdUo2CST0rd1PCEshEd8I6likbc+Nns1Ak5sUqfhLG2pZDM1N8TGY2MGUeB7YwoDs2iNxX/8zo phld+JlSIldsvihMJcGYTP8mfaE5Qzm2hDIt7K2EDamDG06JRuCt/jyMmeVb2Lqnt3Xqld53EU4QiO4RQ8uIQa3EIdGsBgAM/wCm+OdF6cd+dj3lpw8plD+APn8wcU7o2q</latexit>z4 <latexit sha1_base64="GDL7BDHOdr32z1dkxvdmjx3Yb5E=">AB6nicbVDLSgNBEOyNrxhfUY9eBoPgKeyKr2PQi8eI5gHJEmYnvcmQ2dlZlaISz7Bi wdFvPpF3vwbJ8keNFrQUFR1090VJIJr47pfTmFpeWV1rbhe2tjc2t4p7+41dZwqhg0Wi1i1A6pRcIkNw43AdqKQRoHAVjC6nvqtB1Sax/LejBP0IzqQPOSMGivdPfbOeuWKW3VnIH+Jl5MK5Kj3yp/dfszSCKVhgmrd8dzE+BlVhjOBk1I31ZhQNqID7FgqaYTaz2anTsiRVfokjJUtachM/TmR0UjrcRTYzoiaoV70puJ/Xic 14aWfcZmkBiWbLwpTQUxMpn+TPlfIjBhbQpni9lbChlRZmw6JRuCt/jyX9I8qXrnVf2tFK7yuMowgEcwjF4cAE1uIE6NIDBAJ7gBV4d4Tw7b87vLXg5DP78AvOxzcWco2r</latexit>z5 <latexit sha1_base64="ZHnLwo37xRG7ecG/MDNfJXlsmbk=">AB6nicbVBNS8NAEJ34WetX1aOXxSJ4KolI9Vj04rGi/YA2lM120i7dbMLuRqihP8GLB 0W8+ou8+W/ctjlo64OBx3szMwLEsG1cd1vZ2V1bX1js7BV3N7Z3dsvHRw2dZwqhg0Wi1i1A6pRcIkNw43AdqKQRoHAVjC6mfqtR1Sax/LBjBP0IzqQPOSMGivdP/WqvVLZrbgzkGXi5aQMOeq90le3H7M0QmYoFp3PDcxfkaV4UzgpNhNSaUjegAO5ZKGqH2s9mpE3JqlT4JY2VLGjJTf09kNJ6HAW2M6JmqBe9qfif10l NeOVnXCapQcnmi8JUEBOT6d+kzxUyI8aWUKa4vZWwIVWUGZtO0YbgLb68TJrnFa9ace8uyrXrPI4CHMJnIEHl1CDW6hDAxgM4Ble4c0Rzovz7nzMW1ecfOYI/sD5/AEX9o2s</latexit>z6 <latexit sha1_base64="mVKnGAb5gjFGnPRG0Ry1FiSU54=">AB6nicbVBNS8NAEJ34WetX1aOXxSJ4KomI9Vj04rGi/YA2lM120i7dbMLuRqihP8GLB 0W8+ou8+W/ctjlo64OBx3szMwLEsG1cd1vZ2V1bX1js7BV3N7Z3dsvHRw2dZwqhg0Wi1i1A6pRcIkNw43AdqKQRoHAVjC6mfqtR1Sax/LBjBP0IzqQPOSMGivdP/WqvVLZrbgzkGXi5aQMOeq90le3H7M0QmYoFp3PDcxfkaV4UzgpNhNSaUjegAO5ZKGqH2s9mpE3JqlT4JY2VLGjJTf09kNJ6HAW2M6JmqBe9qfif10l NeOVnXCapQcnmi8JUEBOT6d+kzxUyI8aWUKa4vZWwIVWUGZtO0YbgLb68TJrnFe+y4t5dlGvXeRwFOIYTOAMPqlCDW6hDAxgM4Ble4c0Rzovz7nzMW1ecfOYI/sD5/AEZeo2t</latexit>z7 <latexit sha1_base64="GDL7BDHOdr32z1dkxvdmjx3Yb5E=">AB6nicbVDLSgNBEOyNrxhfUY9eBoPgKeyKr2PQi8eI5gHJEmYnvcmQ2dlZlaISz7Bi wdFvPpF3vwbJ8keNFrQUFR1090VJIJr47pfTmFpeWV1rbhe2tjc2t4p7+41dZwqhg0Wi1i1A6pRcIkNw43AdqKQRoHAVjC6nvqtB1Sax/LejBP0IzqQPOSMGivdPfbOeuWKW3VnIH+Jl5MK5Kj3yp/dfszSCKVhgmrd8dzE+BlVhjOBk1I31ZhQNqID7FgqaYTaz2anTsiRVfokjJUtachM/TmR0UjrcRTYzoiaoV70puJ/Xic 14aWfcZmkBiWbLwpTQUxMpn+TPlfIjBhbQpni9lbChlRZmw6JRuCt/jyX9I8qXrnVf2tFK7yuMowgEcwjF4cAE1uIE6NIDBAJ7gBV4d4Tw7b87vLXg5DP78AvOxzcWco2r</latexit>z5 <latexit sha1_base64="ZHnLwo37xRG7ecG/MDNfJXlsmbk=">AB6nicbVBNS8NAEJ34WetX1aOXxSJ4KolI9Vj04rGi/YA2lM120i7dbMLuRqihP8GLB 0W8+ou8+W/ctjlo64OBx3szMwLEsG1cd1vZ2V1bX1js7BV3N7Z3dsvHRw2dZwqhg0Wi1i1A6pRcIkNw43AdqKQRoHAVjC6mfqtR1Sax/LBjBP0IzqQPOSMGivdP/WqvVLZrbgzkGXi5aQMOeq90le3H7M0QmYoFp3PDcxfkaV4UzgpNhNSaUjegAO5ZKGqH2s9mpE3JqlT4JY2VLGjJTf09kNJ6HAW2M6JmqBe9qfif10l NeOVnXCapQcnmi8JUEBOT6d+kzxUyI8aWUKa4vZWwIVWUGZtO0YbgLb68TJrnFa9ace8uyrXrPI4CHMJnIEHl1CDW6hDAxgM4Ble4c0Rzovz7nzMW1ecfOYI/sD5/AEX9o2s</latexit>z6 <latexit sha1_base64="KvmtD2Qzol9BIWTg7UGnEKQGWco=">AB6nicbVBNS8NAEJ3Ur1q/qh69LBbBU0lE1GPRi8eK9gPaUDbTbt0swm7E6G/gQvH hTx6i/y5r9x2+agrQ8GHu/NMDMvSKQw6LrfTmFldW19o7hZ2tre2d0r7x80TZxqxhslrFuB9RwKRvoEDJ24nmNAokbwWjm6nfeuTaiFg94DjhfkQHSoSCUbTS/VP65UrbtWdgSwTLycVyFHvlb+6/ZilEVfIJDWm47kJ+hnVKJjk1I3NTyhbEQHvGOpohE3fjY7dUJOrNInYaxtKSQz9fdERiNjxlFgOyOKQ7PoTcX/vE6 K4ZWfCZWkyBWbLwpTSTAm079JX2jOUI4toUwLeythQ6opQ5tOyYbgLb68TJpnVe+i6t6dV2rXeRxFOIJjOAUPLqEGt1CHBjAYwDO8wpsjnRfn3fmYtxacfOYQ/sD5/AEQYo2n</latexit>z1 <latexit sha1_base64="NzBeE9vq5mC9DhZG+vA9j3WAOK8=">AB6nicbVBNS8NAEJ34WetX1aOXxSJ4KkR9Vj04rGi/YA2lM120y7dbMLuRKihP8GLB 0W8+ou8+W/ctjlo64OBx3szMwLEikMu63s7K6tr6xWdgqbu/s7u2XDg6bJk414w0Wy1i3A2q4FIo3UKDk7URzGgWSt4LRzdRvPXJtRKwecJxwP6IDJULBKFrp/qlX7ZXKbsWdgSwTLydlyFHvlb6/ZilEVfIJDWm47kJ+hnVKJjk2I3NTyhbEQHvGOpohE3fjY7dUJOrdInYaxtKSQz9fdERiNjxlFgOyOKQ7PoTcX/vE6 K4ZWfCZWkyBWbLwpTSTAm079JX2jOUI4toUwLeythQ6opQ5tO0YbgLb68TJrVindRce/Oy7XrPI4CHMJnIEHl1CDW6hDAxgM4Ble4c2Rzovz7nzMW1ecfOYI/sD5/AER5o2o</latexit>z2 <latexit sha1_base64="EFoKUrMUVpSXFxYRkZwK543aNE=">AB6nicbVBNS8NAEJ3Ur1q/qh69LBbBU0lU1GPRi8eK9gPaUDbSbt0swm7G6G/gQvH hTx6i/y5r9x2+ag1QcDj/dmJkXJIJr47pfTmFpeWV1rbhe2tjc2t4p7+41dZwqhg0Wi1i1A6pRcIkNw43AdqKQRoHAVjC6nvqtB1Sax/LejBP0IzqQPOSMGivdPfZOe+WKW3VnIH+Jl5MK5Kj3yp/dfszSCKVhgmrd8dzE+BlVhjOBk1I31ZhQNqID7FgqaYTaz2anTsiRVfokjJUtachM/TmR0UjrcRTYzoiaoV70puJ/Xic 14aWfcZmkBiWbLwpTQUxMpn+TPlfIjBhbQpni9lbChlRZmw6JRuCt/jyX9I8qXrnVf2rFK7yuMowgEcwjF4cAE1uIE6NIDBAJ7gBV4d4Tw7b87vLXg5DP78AvOxzcTao2p</latexit>z3 <latexit sha1_base64="gb5lGORDGwF6MLqPq9ofeWi6bHo=">AB6nicbVBNS8NAEJ3Ur1q/qh69LBbBU0lE1GPRi8eK9gPaUDbTbt0swm7E6G/gQvH hTx6i/y5r9x2+agrQ8GHu/NMDMvSKQw6LrfTmFldW19o7hZ2tre2d0r7x80TZxqxhslrFuB9RwKRvoEDJ24nmNAokbwWjm6nfeuTaiFg94DjhfkQHSoSCUbTS/VPvFeuFV3BrJMvJxUIEe9V/7q9mOWRlwhk9SYjucm6GdUo2CST0rd1PCEshEd8I6likbc+Nns1Ak5sUqfhLG2pZDM1N8TGY2MGUeB7YwoDs2iNxX/8zo phld+JlSIldsvihMJcGYTP8mfaE5Qzm2hDIt7K2EDamDG06JRuCt/jyMmeVb2Lqnt3Xqld53EU4QiO4RQ8uIQa3EIdGsBgAM/wCm+OdF6cd+dj3lpw8plD+APn8wcU7o2q</latexit>z4 <latexit sha1_base64="GDL7BDHOdr32z1dkxvdmjx3Yb5E=">AB6nicbVDLSgNBEOyNrxhfUY9eBoPgKeyKr2PQi8eI5gHJEmYnvcmQ2dlZlaISz7Bi wdFvPpF3vwbJ8keNFrQUFR1090VJIJr47pfTmFpeWV1rbhe2tjc2t4p7+41dZwqhg0Wi1i1A6pRcIkNw43AdqKQRoHAVjC6nvqtB1Sax/LejBP0IzqQPOSMGivdPfbOeuWKW3VnIH+Jl5MK5Kj3yp/dfszSCKVhgmrd8dzE+BlVhjOBk1I31ZhQNqID7FgqaYTaz2anTsiRVfokjJUtachM/TmR0UjrcRTYzoiaoV70puJ/Xic 14aWfcZmkBiWbLwpTQUxMpn+TPlfIjBhbQpni9lbChlRZmw6JRuCt/jyX9I8qXrnVf2tFK7yuMowgEcwjF4cAE1uIE6NIDBAJ7gBV4d4Tw7b87vLXg5DP78AvOxzcWco2r</latexit>z5 <latexit sha1_base64="cwvsV9kZQYfjKp7at+GM3y72gls=">AB6nicbVBNS8NAEJ3Ur1q/qh69LBbBU0lE1GPRi8eK1hbaUDbTbt0swm7EyGE/gQvHhTx6i/y5r9x2+agrQ8GHu/NMDMvSKQw 6LrfTmldW19o7xZ2dre2d2r7h8mjVjLdYLGPdCajhUijeQoGSdxLNaRI3g7GN1O/cS1EbF6wCzhfkSHSoSCUbTSfdb3+tWaW3dnIMvEK0gNCjT71a/eIGZpxBUySY3pem6Cfk41Cib5pNJLDU8oG9Mh71qaMSNn89OnZATqwxIGtbCslM/T2R08iYLApsZ0RxZBa9qfif10xvPJzoZIUuWLzRWEqCcZk+jcZCM0ZyswSyrSwtxI2opoytOlUbAje4svL5PGs7l3U3bvzWuO6iKMR3AMp+DBJTgFp rQAgZDeIZXeHOk8+K8Ox/z1pJTzBzCHzifPw7cjaY=</latexit>y1 <latexit sha1_base64="GZ+Ubp29DwseIYaeWvk9dZQ5OE=">AB6nicbVBNS8NAEJ3Ur1q/qh69LBbBU0mKqMeiF48V7Qe0oWy2k3bpZhN2N0Io/QlePCji1V/kzX/jts1BWx8MPN6bYWZekAiu jet+O4W19Y3NreJ2aWd3b/+gfHjU0nGqGDZLGLVCahGwSU2DTcCO4lCGgUC28H4dua3n1BpHstHkyXoR3QoecgZNVZ6yPq1frniVt05yCrxclKBHI1+as3iFkaoTRMUK27npsYf0KV4UzgtNRLNSaUjekQu5ZKGqH2J/NTp+TMKgMSxsqWNGSu/p6Y0EjrLApsZ0TNSC97M/E/r5ua8NqfcJmkBiVbLApTQUxMZn+TAVfIjMgsoUxeythI6oMzadkg3BW35lbRqVe+y6t5fVOo3eRxFOIFTOAcPrqAOd9 CAJjAYwjO8wpsjnBfn3flYtBacfOY/sD5/AEQYI2n</latexit>y2 <latexit sha1_base64="PmgJ+AknyHpizZRrmqTL6fkvnE=">AB6nicbVBNS8NAEJ3Ur1q/qh69LBbBU0lU1GPRi8eK9gPaUDbTbt0swm7E6GE/gQvHhTx6i/y5r9x2+agrQ8GHu/NMDMvSKQw 6LrfTmFldW19o7hZ2tre2d0r7x80TZxqxhslrFuB9RwKRvoEDJ24nmNAokbwWj26nfeuLaiFg94jhfkQHSoSCUbTSw7h3itX3Ko7A1kmXk4qkKPeK391+zFLI6QSWpMx3MT9DOqUTDJ6VuanhC2YgOeMdSRSNu/Gx26oScWKVPwljbUkhm6u+JjEbGjKPAdkYUh2bRm4r/eZ0Uw2s/EypJkSs2XxSmkmBMpn+TvtCcoRxbQpkW9lbChlRThjadkg3BW3x5mTPqt5l1b2/qNRu8jiKcATHcAoeXEN7q AODWAwgGd4hTdHOi/Ou/Mxby04+cwh/IHz+QMR5I2o</latexit>y3 <latexit sha1_base64="vjgC1/maYdRVBNhrUezoI72Jpvo=">AB6nicbVBNS8NAEJ3Ur1q/qh69LBbBU0mkqMeiF48V7Qe0oWy2k3bpZhN2N0Io/QlePCji1V/kzX/jts1BWx8MPN6bYWZekAiu jet+O4W19Y3NreJ2aWd3b/+gfHjU0nGqGDZLGLVCahGwSU2DTcCO4lCGgUC28H4dua3n1BpHstHkyXoR3QoecgZNVZ6yPq1frniVt05yCrxclKBHI1+as3iFkaoTRMUK27npsYf0KV4UzgtNRLNSaUjekQu5ZKGqH2J/NTp+TMKgMSxsqWNGSu/p6Y0EjrLApsZ0TNSC97M/E/r5ua8NqfcJmkBiVbLApTQUxMZn+TAVfIjMgsoUxeythI6oMzadkg3BW35lbQuqt5l1b2vVeo3eRxFOIFTOAcPrqAOd9 CAJjAYwjO8wpsjnBfn3flYtBacfOY/sD5/AETaI2p</latexit>y4 <latexit sha1_base64="0epzTrGbClv12cWfL2VvsGSTbug=">AB6nicbVDLSgNBEOyNrxhfUY9eBoPgKeyKr2PQi8eI5gHJEmYns8mQ2dlplcISz7BiwdFvPpF3vwbJ8keNLGgoajqprsrSKQw 6LrfTmFldW19o7hZ2tre2d0r7x80TZxqxhslrFuB9RwKRvoEDJ24nmNAokbwWj26nfeuLaiFg94jhfkQHSoSCUbTSw7h30StX3Ko7A1kmXk4qkKPeK391+zFLI6QSWpMx3MT9DOqUTDJ6VuanhC2YgOeMdSRSNu/Gx26oScWKVPwljbUkhm6u+JjEbGjKPAdkYUh2bRm4r/eZ0Uw2s/EypJkSs2XxSmkmBMpn+TvtCcoRxbQpkW9lbChlRThjadkg3BW3x5mTPqt5l1b0/r9Ru8jiKcATHcAoeXEN7q AODWAwgGd4hTdHOi/Ou/Mxby04+cwh/IHz+QMU7I2q</latexit>y5 <latexit sha1_base64="cwvsV9kZQYfjKp7at+GM3y72gls=">AB6nicbVBNS8NAEJ3Ur1q/qh69LBbBU0lE1GPRi8eK1hbaUDbTbt0swm7EyGE/gQvHhTx6i/y5r9x2+agrQ8GHu/NMDMvSKQw 6LrfTmldW19o7xZ2dre2d2r7h8mjVjLdYLGPdCajhUijeQoGSdxLNaRI3g7GN1O/cS1EbF6wCzhfkSHSoSCUbTSfdb3+tWaW3dnIMvEK0gNCjT71a/eIGZpxBUySY3pem6Cfk41Cib5pNJLDU8oG9Mh71qaMSNn89OnZATqwxIGtbCslM/T2R08iYLApsZ0RxZBa9qfif10xvPJzoZIUuWLzRWEqCcZk+jcZCM0ZyswSyrSwtxI2opoytOlUbAje4svL5PGs7l3U3bvzWuO6iKMR3AMp+DBJTgFp rQAgZDeIZXeHOk8+K8Ox/z1pJTzBzCHzifPw7cjaY=</latexit>y1 <latexit sha1_base64="GZ+Ubp29DwseIYaeWvk9dZQ5OE=">AB6nicbVBNS8NAEJ3Ur1q/qh69LBbBU0mKqMeiF48V7Qe0oWy2k3bpZhN2N0Io/QlePCji1V/kzX/jts1BWx8MPN6bYWZekAiu jet+O4W19Y3NreJ2aWd3b/+gfHjU0nGqGDZLGLVCahGwSU2DTcCO4lCGgUC28H4dua3n1BpHstHkyXoR3QoecgZNVZ6yPq1frniVt05yCrxclKBHI1+as3iFkaoTRMUK27npsYf0KV4UzgtNRLNSaUjekQu5ZKGqH2J/NTp+TMKgMSxsqWNGSu/p6Y0EjrLApsZ0TNSC97M/E/r5ua8NqfcJmkBiVbLApTQUxMZn+TAVfIjMgsoUxeythI6oMzadkg3BW35lbRqVe+y6t5fVOo3eRxFOIFTOAcPrqAOd9 CAJjAYwjO8wpsjnBfn3flYtBacfOY/sD5/AEQYI2n</latexit>y2 <latexit sha1_base64="PmgJ+AknyHpizZRrmqTL6fkvnE=">AB6nicbVBNS8NAEJ3Ur1q/qh69LBbBU0lU1GPRi8eK9gPaUDbTbt0swm7E6GE/gQvHhTx6i/y5r9x2+agrQ8GHu/NMDMvSKQw 6LrfTmFldW19o7hZ2tre2d0r7x80TZxqxhslrFuB9RwKRvoEDJ24nmNAokbwWj26nfeuLaiFg94jhfkQHSoSCUbTSw7h3itX3Ko7A1kmXk4qkKPeK391+zFLI6QSWpMx3MT9DOqUTDJ6VuanhC2YgOeMdSRSNu/Gx26oScWKVPwljbUkhm6u+JjEbGjKPAdkYUh2bRm4r/eZ0Uw2s/EypJkSs2XxSmkmBMpn+TvtCcoRxbQpkW9lbChlRThjadkg3BW3x5mTPqt5l1b2/qNRu8jiKcATHcAoeXEN7q AODWAwgGd4hTdHOi/Ou/Mxby04+cwh/IHz+QMR5I2o</latexit>y3 <latexit sha1_base64="vjgC1/maYdRVBNhrUezoI72Jpvo=">AB6nicbVBNS8NAEJ3Ur1q/qh69LBbBU0mkqMeiF48V7Qe0oWy2k3bpZhN2N0Io/QlePCji1V/kzX/jts1BWx8MPN6bYWZekAiu jet+O4W19Y3NreJ2aWd3b/+gfHjU0nGqGDZLGLVCahGwSU2DTcCO4lCGgUC28H4dua3n1BpHstHkyXoR3QoecgZNVZ6yPq1frniVt05yCrxclKBHI1+as3iFkaoTRMUK27npsYf0KV4UzgtNRLNSaUjekQu5ZKGqH2J/NTp+TMKgMSxsqWNGSu/p6Y0EjrLApsZ0TNSC97M/E/r5ua8NqfcJmkBiVbLApTQUxMZn+TAVfIjMgsoUxeythI6oMzadkg3BW35lbQuqt5l1b2vVeo3eRxFOIFTOAcPrqAOd9 CAJjAYwjO8wpsjnBfn3flYtBacfOY/sD5/AETaI2p</latexit>y4 <latexit sha1_base64="cwvsV9kZQYfjKp7at+GM3y72gls=">AB6nicbVBNS8NAEJ3Ur1q/qh69LBbBU0lE1GPRi8eK1hbaUDbTbt0swm7EyGE/gQvHhTx6i/y5r9x2+agrQ8GHu/NMDMvSKQw 6LrfTmldW19o7xZ2dre2d2r7h8mjVjLdYLGPdCajhUijeQoGSdxLNaRI3g7GN1O/cS1EbF6wCzhfkSHSoSCUbTSfdb3+tWaW3dnIMvEK0gNCjT71a/eIGZpxBUySY3pem6Cfk41Cib5pNJLDU8oG9Mh71qaMSNn89OnZATqwxIGtbCslM/T2R08iYLApsZ0RxZBa9qfif10xvPJzoZIUuWLzRWEqCcZk+jcZCM0ZyswSyrSwtxI2opoytOlUbAje4svL5PGs7l3U3bvzWuO6iKMR3AMp+DBJTgFp rQAgZDeIZXeHOk8+K8Ox/z1pJTzBzCHzifPw7cjaY=</latexit>y1 <latexit sha1_base64="GZ+Ubp29DwseIYaeWvk9dZQ5OE=">AB6nicbVBNS8NAEJ3Ur1q/qh69LBbBU0mKqMeiF48V7Qe0oWy2k3bpZhN2N0Io/QlePCji1V/kzX/jts1BWx8MPN6bYWZekAiu jet+O4W19Y3NreJ2aWd3b/+gfHjU0nGqGDZLGLVCahGwSU2DTcCO4lCGgUC28H4dua3n1BpHstHkyXoR3QoecgZNVZ6yPq1frniVt05yCrxclKBHI1+as3iFkaoTRMUK27npsYf0KV4UzgtNRLNSaUjekQu5ZKGqH2J/NTp+TMKgMSxsqWNGSu/p6Y0EjrLApsZ0TNSC97M/E/r5ua8NqfcJmkBiVbLApTQUxMZn+TAVfIjMgsoUxeythI6oMzadkg3BW35lbRqVe+y6t5fVOo3eRxFOIFTOAcPrqAOd9 CAJjAYwjO8wpsjnBfn3flYtBacfOY/sD5/AEQYI2n</latexit>y2 <latexit sha1_base64="PmgJ+AknyHpizZRrmqTL6fkvnE=">AB6nicbVBNS8NAEJ3Ur1q/qh69LBbBU0lU1GPRi8eK9gPaUDbTbt0swm7E6GE/gQvHhTx6i/y5r9x2+agrQ8GHu/NMDMvSKQw 6LrfTmFldW19o7hZ2tre2d0r7x80TZxqxhslrFuB9RwKRvoEDJ24nmNAokbwWj26nfeuLaiFg94jhfkQHSoSCUbTSw7h3itX3Ko7A1kmXk4qkKPeK391+zFLI6QSWpMx3MT9DOqUTDJ6VuanhC2YgOeMdSRSNu/Gx26oScWKVPwljbUkhm6u+JjEbGjKPAdkYUh2bRm4r/eZ0Uw2s/EypJkSs2XxSmkmBMpn+TvtCcoRxbQpkW9lbChlRThjadkg3BW3x5mTPqt5l1b2/qNRu8jiKcATHcAoeXEN7q AODWAwgGd4hTdHOi/Ou/Mxby04+cwh/IHz+QMR5I2o</latexit>y3 <latexit sha1_base64="vjgC1/maYdRVBNhrUezoI72Jpvo=">AB6nicbVBNS8NAEJ3Ur1q/qh69LBbBU0mkqMeiF48V7Qe0oWy2k3bpZhN2N0Io/QlePCji1V/kzX/jts1BWx8MPN6bYWZekAiu jet+O4W19Y3NreJ2aWd3b/+gfHjU0nGqGDZLGLVCahGwSU2DTcCO4lCGgUC28H4dua3n1BpHstHkyXoR3QoecgZNVZ6yPq1frniVt05yCrxclKBHI1+as3iFkaoTRMUK27npsYf0KV4UzgtNRLNSaUjekQu5ZKGqH2J/NTp+TMKgMSxsqWNGSu/p6Y0EjrLApsZ0TNSC97M/E/r5ua8NqfcJmkBiVbLApTQUxMZn+TAVfIjMgsoUxeythI6oMzadkg3BW35lbQuqt5l1b2vVeo3eRxFOIFTOAcPrqAOd9 CAJjAYwjO8wpsjnBfn3flYtBacfOY/sD5/AETaI2p</latexit>y4 Figure 1: Comparison between (a) cascaded pipeline, (b) direct simultaneous ST, and (c) our ASR-assisted simultaneous ST. …

Content truncated to 500 words

Labeled Text

Simultaneous speech-to-text translation is widely useful in many scenarios. The conventional cascaded approach uses a pipeline of streaming ASR followed by simultaneous MT, but suffers from error propagation and extra latency. To alleviate these issues, recent efforts attempt to directly translate the source speech into target text simultaneously, but this is much harder due to the combination of two separate tasks. We instead propose a new paradigm with the advantages of both cascaded and endto-end approaches. The key idea is to use two separate, but synchronized, decoders on streaming ASR and direct speech-to-text translation (ST), respectively, and the intermediate results of ASR guide the decoding policy of (but is not fed as input to) ST. During training time, we use multitask learning to jointly learn these two tasks with a shared encoder. En-toDe and En-to-Es experiments on the MuSTC dataset demonstrate that our proposed technique achieves substantially better translation quality at similar levels of latency. 1

Publication Details

Paper ID:
33778
Year:
2021
Publication:
TIPSTER
Publication Type:
conf
DOI:
nan

Links

Paper URL PDF Publication URL