Continual Knowledge Distillation for Neural Machine Translation

Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics Volume 1: Long Papers, pages 7978–7996 July 9-14, 2023 ©2023 Association for Computational Linguistics Continual Knowledge Distillation for Neural Machine Translation Yuanchi Zhang1,3, Peng Li∗2,4, Maosong Sun1,3, Yang Liu∗ 1,2,3,4 1Dept. of Comp. Sci. & Tech., Institute for AI, Tsinghua University, Beijing, China 2Institute for AI Industry Research (AIR), Tsinghua University, Beijing, China 3Beijing National Research Center for Information Science and Technology 4Shanghai Artificial Intelligence Laboratory, Shanghai, China yuanchi-21@mails.tsinghua.edu.cn; lipeng@air.tsinghua.edu.cn {sms,liuyang2011}@tsinghua.edu.cn Abstract While many parallel corpora are not publicly accessible for data copyright, data privacy and competitive differentiation reasons, trained translation models are increasingly available on open platforms. In this work, we propose a method called continual knowledge distilla- tion to take advantage of existing translation models to improve one model of interest. The basic idea is to sequentially transfer knowledge from each trained model to the distilled model. Extensive experiments on Chinese-English and German-English datasets show that our method achieves significant and consistent improve- ments over strong baselines under both homo- geneous and heterogeneous trained model set- tings and is robust to malicious models.1 1 Introduction Current neural machine translation (NMT) systems often face such a situation: parallel corpora are not publicly accessible but trained models are more readily available. On the one hand, many data owners are usually unwilling to share their par- allel corpora with the public for data copyright, data privacy and competitive differentiation rea- sons, leading to recent interests in federated learn- ing for NMT (Wang et al., 2021b; Roosta et al., 2021). On the other hand, trained NMT models are increasingly available on platforms such as Huggin- face (https://huggingface.co) and Opus-MT (https://opus.nlpl.eu/Opus-MT) since these models can be directly used without public access to the original training data. As a result, a question naturally arises: can we take advantage of increasingly available trained NMT models to enhance one NMT model of inter- est? In this work, we propose a method called Con- tinual Knowledge Distillation (CKD) to address ∗Corresponding authors. 1The source code is available at https://github.com/ THUNLP-MT/CKD. <latexit sha1_base64="tAH4Y/rWmAp4jYQLNhzDQBjy6r0=">ACnicbVFdS8MwFM3q15xfUx9CQ7Bp9 GKX49DQUQRJzgdrEPSLJ1hSVqSW3GUPvkDfNWf5r8x3Yq6zQuBw7n3JzkBrHgBlz3q+TMzS8sLpWXKyura+sb1c2tBxMlmrIWjUSk2wExTHDFWsBsHasGZGBYI/B4DzvP74wbXik7mEYs64kfcVDTglYquX3IjBP1Zpbd0eFZ 4FXgBoqvm0WXqzRpIpoAKYkzHc2PopkQDp4JlFT8xLCZ0QPqsY6EikpluOkqb4T3L9HAYaXsU4BH715ESacxQBlYpCTyb6V5O/tfrJBCedlOu4gSYouOLwkRgiHD+dNzjmlEQwsI1dxmxfSZaELBftDELedX1/mwbJqME0Vh hlWJDTtBjKrVPweC32i+5K8pn6eNopTX0s8prJs71fB1YyCqx/F9IDCbdfmTS9pFjwc1L3j+tHdYa1xViywjHbQLtpHjpBDXSJmqiFKOLoHX2gTwc7F86NczuWOqXCs40myml/A13S0Pg=</latexit>. . . <latexit sha1_base64="tAH4Y/rWmAp4jYQLNhzDQBjy6r0=">ACnicbVFdS8MwFM3q15xfUx9CQ7Bp9 GKX49DQUQRJzgdrEPSLJ1hSVqSW3GUPvkDfNWf5r8x3Yq6zQuBw7n3JzkBrHgBlz3q+TMzS8sLpWXKyura+sb1c2tBxMlmrIWjUSk2wExTHDFWsBsHasGZGBYI/B4DzvP74wbXik7mEYs64kfcVDTglYquX3IjBP1Zpbd0eFZ 4FXgBoqvm0WXqzRpIpoAKYkzHc2PopkQDp4JlFT8xLCZ0QPqsY6EikpluOkqb4T3L9HAYaXsU4BH715ESacxQBlYpCTyb6V5O/tfrJBCedlOu4gSYouOLwkRgiHD+dNzjmlEQwsI1dxmxfSZaELBftDELedX1/mwbJqME0Vh hlWJDTtBjKrVPweC32i+5K8pn6eNopTX0s8prJs71fB1YyCqx/F9IDCbdfmTS9pFjwc1L3j+tHdYa1xViywjHbQLtpHjpBDXSJmqiFKOLoHX2gTwc7F86NczuWOqXCs40myml/A13S0Pg=</latexit>. . . Teacher Models Student Model <latexit sha1_base64="tAH4Y/rWmAp4jYQLNhzDQBjy6r0=">ACnicbVFdS8MwFM3q15xfUx9CQ7Bp9 GKX49DQUQRJzgdrEPSLJ1hSVqSW3GUPvkDfNWf5r8x3Yq6zQuBw7n3JzkBrHgBlz3q+TMzS8sLpWXKyura+sb1c2tBxMlmrIWjUSk2wExTHDFWsBsHasGZGBYI/B4DzvP74wbXik7mEYs64kfcVDTglYquX3IjBP1Zpbd0eFZ 4FXgBoqvm0WXqzRpIpoAKYkzHc2PopkQDp4JlFT8xLCZ0QPqsY6EikpluOkqb4T3L9HAYaXsU4BH715ESacxQBlYpCTyb6V5O/tfrJBCedlOu4gSYouOLwkRgiHD+dNzjmlEQwsI1dxmxfSZaELBftDELedX1/mwbJqME0Vh hlWJDTtBjKrVPweC32i+5K8pn6eNopTX0s8prJs71fB1YyCqx/F9IDCbdfmTS9pFjwc1L3j+tHdYa1xViywjHbQLtpHjpBDXSJmqiFKOLoHX2gTwc7F86NczuWOqXCs40myml/A13S0Pg=</latexit>. . . <latexit sha1_base64="tAH4Y/rWmAp4jYQLNhzDQBjy6r0=">ACnicbVFdS8MwFM3q15xfUx9CQ7Bp9 GKX49DQUQRJzgdrEPSLJ1hSVqSW3GUPvkDfNWf5r8x3Yq6zQuBw7n3JzkBrHgBlz3q+TMzS8sLpWXKyura+sb1c2tBxMlmrIWjUSk2wExTHDFWsBsHasGZGBYI/B4DzvP74wbXik7mEYs64kfcVDTglYquX3IjBP1Zpbd0eFZ 4FXgBoqvm0WXqzRpIpoAKYkzHc2PopkQDp4JlFT8xLCZ0QPqsY6EikpluOkqb4T3L9HAYaXsU4BH715ESacxQBlYpCTyb6V5O/tfrJBCedlOu4gSYouOLwkRgiHD+dNzjmlEQwsI1dxmxfSZaELBftDELedX1/mwbJqME0Vh hlWJDTtBjKrVPweC32i+5K8pn6eNopTX0s8prJs71fB1YyCqx/F9IDCbdfmTS9pFjwc1L3j+tHdYa1xViywjHbQLtpHjpBDXSJmqiFKOLoHX2gTwc7F86NczuWOqXCs40myml/A13S0Pg=</latexit>. . . <latexit sha1_base64="SjxVmf5PxAr3RD9Bow16frC/WPA=">ACpnicbZFda9swFIYVd2s7r+vnZW/EQiBXwS79uizrTWmhdNA0hdiksi InIpKsSsdlwfiqP6K328/av5mcmG5JdkDw8pz3fKCTaMEtBMHvhrf24eP6xuYn/PWl+2d3b39B5vlhrIuzURmHhNimeCKdYGDYI/aMCITwXrJ5LK916YsTxT9zDVLJZkpHjKQGH4iRaTHvBwUA52m0EnmAVeFWEtmqiOu8Fe4zUaZjSXTAEVxNp+GiIC2KAU8FKP8ot04ROyIj1nVREM hsXs61L3HJkiNPMuKcAz+i/FQWR1k5l4pySwNgu5yr4v1w/h/Q8LrjSOTBF54PSXGDIcPUFeMgNoyCmThBquNsV0zExhIL7qIUpl9c3VbNyGepcUVihKnfLsBElr4fDVkaETOS5EcRVdtmuoiMxHNUlq2/Dq5WHFy9O5Yb1NXubOHykVbFw1EnPO2cfD9uXnyrD7iJDtFX1EYhOkMX6ArdoS6 i6Bm9oZ/ol9f2br2u15tbvUZdc4AWwnv6A0xE1L0=</latexit>φt <latexit sha1_base64="Ti3mFPD9Bb21bInb3G7OW2MXMfY=">ACqnicbZHLThsxFIad4VIaboEu7GIkJBA0QxqC0sEm6rd0IpwUSaKPB5PYmF 7LPu4ajSaFY/BFh6Kt6knGRWScCRLv7zn4t8Ei24hTB8aQRLyurH9Y+Ntc3Nre2Wzu71zZ3hrIuzUVubhNimeCKdYGDYLfaMCITwW6S+4sqf/OHGctzdQVjzfqSDBXPOCXg0aC1HSeyiPWIl4MCDqNy0GqHnXASeFEtWijOi4HO42HOM2pk0wBFcTaXhRq6BfEAKeClc3YWaYJvSdD1vNSEclsv5hsXuJ 9T1Kc5cY/BXhC31YURFo7lol3SgIjO5+r4Hu5noPstF9wpR0wRaeDMicw5Lj6BpxywyiIsReEGu53xXREDKHgP2tmysWPn1Wzch5qpygsUOX8sjMwkWzGacsi4kZSvK3iKtc13ERuIpKsv9VwdXCw6u/jvmG9TV/mzR/JEWxfVxJ/rW+frS/vsvD7gGvqM9tABitAJOkPf0SXqIocekRP6Dk4Cn4Hd0F vag0adc0nNBNB+g/6/9Ve</latexit>φt+ 1 <latexit sha1_base64="kFwa7Ofs3KOSi3nLmjTjrb4Zg=">ACqnicbZHLThsxFIad4VIaboEu7GIkFhANIPawhLBpmo3tCJclIkij8 eTWNgeyz6uGo1mxWOwhYfibepJRoUkHMnSr+/85yKfRAtuIQxfGsHS8srqh7WPzfWNza3t1s7utc2doaxLc5Gb24RYJrhiXeAg2K02jMhEsJvk/qLK3/xhxvJcXcFYs74kQ8UzTgl4NGhtx4ksYj3i5aCAo6gctNphJ5wEXhRLdqojsvBTuMhTnPqJFNABbG2F4Ua+gUxwKlgZTN2lmlC78mQ9b xURDLbLyabl3jfkxRnufFPAZ7QtxUFkdaOZeKdksDIzucq+F6u5yA7RdcaQdM0emgzAkMOa6+AafcMApi7AWhvtdMR0RQyj4z5qZcvHjZ9WsnIfaKQoLVDm/7AxMZNlsxinLYmKGkvwt4mrbXBexkXiKynL/1cHVgoOr/475BnW1P1s0f6RFcX3cib51v760j47rw+4hj6jPXSAInSCztB3dI m6iCKHtETeg4Og9/BXdCbWoNGXfMJzUSQ/gP/b9Vg</latexit>φt−1 <latexit sha1_base64="DOUhpxtBToTBelWfUcnHpbh5lNY=">ACrHicbZFNb9NAEIY35qNtykcCRy6rRpUQh8hGQDlW9ILg 0kpNWyl2o/FmnKy6u7Z2x0Bk+cTv4Nr+Jv4N68SCJmGklV4986HdtJCSUdh+LsTPHj46PHO7l53/8nTZ897/RcXLi+twJHIVW6vUnCopMERSVJ4VgEnSq8TG9OmvzlN7RO5uacFgUmGmZGZlIAeTp9eNUVzHNkaCe0HX1p70BuEwXAbfFlErBqyN0m/8zOe5qL UaEgocG4chQUlFViSQmHdjUuHBYgbmOHYSwMaXVItd6/5oSdTnuXWP0N8Se9XVKCdW+jUOzXQ3G3mGvi/3Lik7GNSVOUhEasBmWl4pTz5iP4VFoUpBZegLDS78rFHCwI8t+1NuXky9emWb0Ji9I2qKm9MuwVTX3W48xSwGO9Pwo4qbfOiq3mK1TXh/8c0mw5pP nr2GzQVvuzRZtH2hYXb4fRh+H7s3eD40/tAXfZK3bAXrOIHbFj9pmdshET7Dv7xW7ZXTAMzoNxkKysQaetecnWIsj+AIwW1nM=</latexit>✓⇤ t <latexit sha1_base64="l1zFrIo6E3NhSVGmK41NjRVrZwk=">ACsHicbZHLahsxFIbl6S1b05LV9mImEBpwcyE3pah2ZRk0KdGDzu cEbW2GokzSCdKTVCqz5Jt+0T9W2isYcmtnNA8POd/1zQySspLMbxv0505+69+w92HnYfPX7y9Flv9/m5LWvD+JCVsjSjHCyXQvMhCpR8VBkOKpf8Ir8bvIXP7ixotRfcVHxiYKZFoVgAFlvZdprlyKc47gM4dvEv/NvfZrx8P4mXQbZG0ok/aOMt2O7/SaclqxTUyCdaOk7jCiQODgknu u2lteQXsEmZ8HKQGxe3ELf39CQKS1KE5GuqQ3KxwoaxcqD04FOLebuQbelhvXWHycOKGrGrlmq0FLSmWtPkMOhWGM5SLIAZEXalbA4GIYvW5tyfHLaNPObsKo1wy2q67DsGsyV73bTKS9SMDMFP13abFtWLjWKrpD3B9cOobcQv93bDZoq8PZks0jbYvzw0HyfvDuy9v+0af2gDtk j+yTVyQhH8gR+UzOyJAw4shv8of8jQ6jUZRFsLJGnbmBVmL6PsVMFnX7w=</latexit>✓⇤ t+ 1 <latexit sha1_base64="RgU/L5sxqXIh2sn5BLyby1RPXM8=">ACsHicbZHLahsxFIbl6S1b05LV9mImEAp1MyE3pah2ZRk0KdG DzucEbW2GokzSCdKTVCqz5Jt+0T9W2isYcmtnNA8POd/1zQySspLMbxv0505+69+w92HnYfPX7y9Flv9/m5LWvD+JCVsjSjHCyXQvMhCpR8VBkOKpf8Ir8bvIXP7ixotRfcVHxiYKZFoVgAFlvZdprlyKc47gM4dvEv/NvfZrx8P4mXQbZG0ok/aOMt2O7/SaclqxTUyCdaOk 7jCiQODgknu2lteQXsEmZ8HKQGxe3ELf39CQKS1KE5GuqQ3KxwoaxcqD04FOLebuQbelhvXWHycOKGrGrlmq0FLSmWtPkMOhWGM5SLIAZEXalbA4GIYvW5tyfHLaNPObsKo1wy2q67DsGsyV73bTKS9SMDMFP13abFtWLjWKrpD3B9cOobcQv93bDZoq8PZks0jbYvzw 0HyfvDuy9v+0af2gDtkj+yTVyQhH8gR+UzOyJAw4shv8of8jQ6jUZRFsLJGnbmBVmL6PsVNHX8Q=</latexit>✓⇤ t−1 Figure 1: Continual knowledge distillation for neural machine translation. Knowledge is continually distilled from a sequence of teacher models to one student model. At each time step, the current student model (i.e., ϕt) fuses the knowledge transferred from both the current teacher model (i.e., θ∗ t ) and the previous student model (i.e., ϕt−1). All teacher models are frozen and the student model is trainable. Different models are high- lighted in different colors. this problem for NMT. As shown in Figure 1, we as- sume that multiple trained …

Content truncated to 500 words