Counterfactual Data Augmentation for Neural Machine Translation
20951Authors
nan
Abstract
AbstractWe propose a data augmentation method for neural machine translation. It works by interpreting language models and phrasal alignment causally. Specifically, it creates augmented parallel translation corpora by generating (path-specific) counterfactual aligned phrases. We generate these by sampling new source phrases from a masked language model, then sampling an aligned counterfactual target phrase by noting that a translation language model can be interpreted as a Gumbel-Max Structural Causal Model (Oberst and Sontag, 2019). Compared to previous work, our method takes both context and alignment into account to maintain the symmetry between source and target sequences. Experiments on IWSLT’15 English → Vietnamese, WMT’17 English → German, WMT’18 English → Turkish, and WMT’19 robust English → French show that the method can improve the performance of translation, backtranslation and translation robustness.
Keywords
Full Content
Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 187–197 June 6–11, 2021. ©2021 Association for Computational Linguistics 187 Counterfactual Data Augmentation for Neural Machine Translation Qi Liu‡, Matt J. Kusner†∗, Phil Blunsom‡⋄, ‡University of Oxford ⋄DeepMind †University College London ∗The Alan Turing Institute ‡{firstname.lastname}@cs.ox.ac.uk †m.kusner@ucl.ac.uk Abstract We propose a data augmentation method for neural machine translation. It works by in- terpreting language models and phrasal align- ment causally. Specifically, it creates aug- mented parallel translation corpora by gen- erating (path-specific) counterfactual aligned phrases. We generate these by sampling new source phrases from a masked language model, then sampling an aligned counterfactual tar- get phrase by noting that a translation lan- guage model can be interpreted as a Gumbel- Max Structural Causal Model (Oberst and Sontag, 2019). Compared to previous work, our method takes both context and alignment into account to maintain the symmetry be- tween source and target sequences. Experi- ments on IWSLT’15 English → Vietnamese, WMT’17 English → German, WMT’18 En- glish → Turkish, and WMT’19 robust English → French show that the method can improve the performance of translation, backtranslation and translation robustness. 1 Introduction Neural machine translation (NMT) models (Kalch- brenner and Blunsom, 2013; Bahdanau et al., 2014; Vaswani et al., 2017) have reached state-of-the-art performance on various benchmarks. However, these models frequently rely on large-scale paral- lel corpora for training, exhibiting degraded per- formance on low-resource languages (Zoph et al., 2016). Further, modern NMT systems are often brittle, as noises (e.g. grammatical errors) can cause significant mistranslations (Sakaguchi et al., 2017; Michel and Neubig, 2018). Data augmentation is a promising direction to overcome these issues. It works by enlarging the number of data points for training without manually collecting new data. It is widely used to improve diversity and robustness and to avoid overfitting on small datasets. Even though data augmentation (e.g. image flipping, cropping and blurring) has GX 1 <latexit sha1_base64="k9WkuKGgiaiqCUB1ED1VEkolNs= ">ACGHicbVDLSsNAFJ3UV62vqDvdBIvgqiRV0GXBhS4r2Ae0sUymN+3QySTMTIQSAn6H+BWP8GduHXnF/gbTtIsbOuBgXPuZ d753gRo1LZ9rdRWldW98ob1a2tnd298z9g7YMY0GgRUIWiq6HJTDKoaWoYtCNBODAY9DxJteZ3kEIWnI79U0AjfAI059SrDS0s A86gdYjT0/uUkHiZM+5CXBLOmA7Nq1+wc1jJxClJFBZoD86c/DEkcAFeEYSl7jh0pN8FCUcIgrfRjCREmEzyCnqYcByDdJP9Dap 1qZWj5odCPKytX/04kOJByGni6MztRLnqZ+J/Xi5V/5SaUR7ECTmaL/JhZKrSyQKwhFUAUm2qCiaD6VouMscBE6djmtkidCwhgW TLOYg7LpF2vOe1+t1FtWEXGZXRMTpBZ8hBl6iBblETtRBT+gFvaI349l4Nz6Mz1lryShmDtEcjK9fey2hUA=</latexit> GX 2 <latexit sha1_base64="rhCGUP7ylxcMUtAWVv78y5IyK28= ">ACGHicbVDLSsNAFJ3UV62vqDvdBIvgqiRV0GXBhS4r2Ae0sUymN+3QySTMTIQSAn6H+BWP8GduHXnF/gbTtIsbOuBgXPuZ d753gRo1LZ9rdRWldW98ob1a2tnd298z9g7YMY0GgRUIWiq6HJTDKoaWoYtCNBODAY9DxJteZ3kEIWnI79U0AjfAI059SrDS0s A86gdYjT0/uUkHST19yEuCWdJNB2bVrtk5rGXiFKSKCjQH5k9/GJI4AK4Iw1L2HDtSboKFoRBWunHEiJMJngEPU05DkC6Sf6H1D rVytDyQ6EfV1au/p1IcCDlNPB0Z3aiXPQy8T+vFyv/yk0oj2IFnMwW+TGzVGhlgVhDKoAoNtUE0H1rRYZY4GJ0rHNbZE6FxDAs mScxRyWSbtec85r9buLasMuMiqjY3SCzpCDLlED3aImaiGCntALekVvxrPxbnwYn7PWklHMHKI5GF+/fNOhUQ=</latexit> GX 3 <latexit sha1_base64="GmnjJWgBubJgjXJtnNZcxV6g= ">ACGHicbVDLSsNAFJ34rPUVdaebYBFclaQVdFlwocsK9gFtLJPpTt0MgkzE6GEgN/hB7jVT3Anbt35Bf6GkzQL23pg4Nxz7u XeOV7EqFS2/W2srK6tb2yWtsrbO7t7+bBYVuGsSDQIiELRdfDEhjl0FJUMehGAnDgMeh4k+vM7zyCkDTk92oagRvgEac+JVhpaW Ae9wOsxp6f3KSDpJ4+5CXBLOmA7NiV+0c1jJxClJBZoD86c/DEkcAFeEYSl7jh0pN8FCUcIgLfdjCREmEzyCnqYcByDdJP9Dap 1pZWj5odCPKytX/04kOJByGni6MztRLnqZ+J/Xi5V/5SaUR7ECTmaL/JhZKrSyQKwhFUAUm2qCiaD6VouMscBE6djmtkidCwhgW TLOYg7LpF2rOvVq7e6i0rCLjEroBJ2ic+SgS9RAt6iJWoigJ/SCXtGb8Wy8Gx/G56x1xShmjtAcjK9fnmhUg=</latexit> GX 4 <latexit sha1_base64="RQRWpR4wEM/zCK6MyWqydogiQ+w= ">ACGHicbVDLSsNAFJ34rPUVdaebYBFclaQWdFlwocsK9gFtLJPpTt0MgkzE6GEgN/hB7jVT3Anbt35Bf6GkzQL23pg4Nxz7u XeOV7EqFS2/W2srK6tb2yWtsrbO7t7+bBYVuGsSDQIiELRdfDEhjl0FJUMehGAnDgMeh4k+vM7zyCkDTk92oagRvgEac+JVhpaW Ae9wOsxp6f3KSDpJ4+5CXBLOmA7NiV+0c1jJxClJBZoD86c/DEkcAFeEYSl7jh0pN8FCUcIgLfdjCREmEzyCnqYcByDdJP9Dap 1pZWj5odCPKytX/04kOJByGni6MztRLnqZ+J/Xi5V/5SaUR7ECTmaL/JhZKrSyQKwhFUAUm2qCiaD6VouMscBE6djmtkidCwhgW TLOYg7LpF2rOhfV2l290rCLjEroBJ2ic+SgS9RAt6iJWoigJ/SCXtGb8Wy8Gx/G56x1xShmjtAcjK9fgB+hUw=</latexit> i would love a sandwich ich würde ein sandwich lieben GY 1 <latexit sha1_base64="z1XM2OhTgc51FaxMG+3wlSz3zs= ">ACGHicbVDLSsNAFJ34rPUVdaebwSK4KkVdFlwocsK9iFtDJPpTt08mBmIpQ8Dv8ALf6Ce7ErTu/wN9w0mZhWw8MnHvOvd w7x4s5k8qyvo2l5ZXVtfXSRnlza3tn19zb8koERSaNOKR6HhEAmchNBVTHDqxABJ4HNre6Cr3248gJIvCOzWOwQnIGQ+o0RpyT UPewFRQ89PrzM3tbOHSUkJT+8z16xYVWsCvEjsglRQgYZr/vT6EU0CBXlRMqubcXKSYlQjHLIyr1EQkzoiAygq2lIApBOvlDhk +0sd+JPQLFZ6ofydSEkg5DjzdmZ8o571c/M/rJsq/dFIWxomCkE4X+QnHKsJ5ILjPBFDFx5oQKpi+FdMhEYQqHdvMFqlzAQE8T 8aez2GRtGpV+6xauz2v1K0ioxI6QsfoFNnoAtXRDWqgJqLoCb2gV/RmPBvxofxOW1dMoqZAzQD4+sXfMehUQ=</latexit> GY 3 <latexit sha1_base64="WCQxaEeNXu9Aufbl2yA+fYO1yGg= ">ACGHicbVDLSsNAFJ3UV62vqDvdBIvgqiStoMuC1WsA9pY5hMb9qhkwczE6GEgN/hB7jVT3Anbt35Bf6GkzQL23pg4Nxz7u XeOW7EqJCm+a2VlbX1jfKm5Wt7Z3dPX3/oCPCmBNok5CFvOdiAYwG0JZUMuhFHLDvMui6k6vM7z4CFzQM7uQ0AtvHo4B6lGCpJE c/GvhYjl0vuU6dpJE+5CXBLlPHb1q1swcxjKxClJFBVqO/jMYhiT2IZCEYSH6lhlJO8FcUsIgrQxiAREmEzyCvqIB9kHYSf6H1D hVytDwQq5eI1c/TuRYF+Iqe+qzuxEsehl4n9eP5bepZ3QIolBGS2yIuZIUMjC8QYUg5EsqkimHCqbjXIGHNMpIptbotQuQAHl iVjLeawTDr1mtWo1W/Pq02zyKiMjtEJOkMWukBNdINaqI0IekIv6BW9ac/au/ahfc5aS1oxc4jmoH39AoAToVM=</latexit> GY 4 <latexit sha1_base64="V4NqcQT+xKxlby5wn7lpUAxZg8Q= ">ACGHicbVDLSsNAFJ3UV62vqDvdBIvgqiS1oMuC1WsA9pY5hMb9qhkwczE6GEgN/hB7jVT3Anbt35Bf6GkzQL23pg4Nxz7u XeOW7EqJCm+a2VlbX1jfKm5Wt7Z3dPX3/oCPCmBNok5CFvOdiAYwG0JZUMuhFHLDvMui6k6vM7z4CFzQM7uQ0AtvHo4B6lGCpJE c/GvhYjl0vuU6dpJE+5CXBLlPHb1q1swcxjKxClJFBVqO/jMYhiT2IZCEYSH6lhlJO8FcUsIgrQxiAREmEzyCvqIB9kHYSf6H1D hVytDwQq5eI1c/TuRYF+Iqe+qzuxEsehl4n9eP5bepZ3QIolBGS2yIuZIUMjC8QYUg5EsqkimHCqbjXIGHNMpIptbotQuQAHl iVjLeawTDr1mnVeq982qk2zyKiMjtEJOkMWukBNdINaqI0IekIv6BW9ac/au/ahfc5aS1oxc4jmoH39AoG5oVQ=</latexit> GY 5 <latexit sha1_base64="V7US2RzZJ4vJ3zHJ61FCV1eBSc4= ">ACGHicbVDLSsNAFJ3UV62vqDvdBIvgqiRV0WXBhS4r2Ie0MUymN+3QyYOZiVBCwO/wA9zqJ7gTt+78An/DSZqFbT0wcO4593 LvHDdiVEjT/NZKS8srq2vl9crG5tb2jr671xZhzAm0SMhC3nWxAEYDaEkqGXQjDth3GXTc8VXmdx6BCxoGd3ISge3jYUA9SrBUkq Mf9H0sR6XKdOcp4+5CXBLlPHb1q1swcxiKxClJFBZqO/tMfhCT2IZCEYSF6lhlJO8FcUsIgrfRjAREmYzyEnqIB9kHYSf6H1D hWysDwQq5eI1c/TuRYF+Iie+qzuxEMe9l4n9eL5bepZ3QIolBGS6yIuZIUMjC8QYUA5EsokimHCqbjXICHNMpIptZotQuQAHl iVjzewSNr1mnVaq9+eVRtmkVEZHaIjdIsdIEa6AY1UQsR9IRe0Ct60561d+1D+5y2lrRiZh/NQPv6BYNfoVU=</latexit> GX 5 <latexit sha1_base64="/6YisMcD2nY5Ql5gU7jQTisf+Y= ">ACGHicbVDLSsNAFJ34rPUVdaebYBFclaQquiy40GUF+4A2lsn0ph06mYSZiVBCwO/wA9zqJ7gTt+78An/DSZqFbT0wcO4593 LvHC9iVCrb/jaWldW19ZLG+XNre2dXNvyXDWBokpCFouNhCYxyaCqGHQiATjwGLS98Xmtx9BSBryezWJwA3wkFOfEqy01D cPewFWI89PbtJ+cpE+5CXBLOmkfbNiV+0c1iJxClJBRp986c3CEkcAFeEYSm7jh0pN8FCUcIgLfdiCREmYzyErqYcByDdJP9Dap 1oZWD5odCPKytX/04kOJByEni6MztRznuZ+J/XjZV/5SaUR7ECTqaL/JhZKrSyQKwBFUAUm2iCiaD6VouMsMBE6dhmtkidCwhgW TLOfA6LpFWrOmfV2t15pW4XGZXQETpGp8hBl6iOblEDNRFBT+gFvaI349l4Nz6Mz2nrklHMHKAZGF+/gcWhVA=</latexit> X <latexit sha1_base64="DSqt/3Qa7/Ug0Q+grni0IN8OAF8= ">ACB3icbVDLSgMxFM34rPVdekmWARXZaYKuiy6cVnBPmA6lEx6pw3NJEOSEcrQD/AD3OonuBO3foZf4G+YaWdhWw8EDufcyz 05YcKZNq7aytb2xubZd2yrt7+weHlaPjtpapotCikvVDYkGzgS0DMcuokCEocOuH4Lvc7T6A0k+LRTBIYjIULGKUGCv5vZ iYESU8607lapbc2fAq8QrSBUVaPYrP72BpGkMwlBOtPY9NzFBRpRhlMO03Es1JISOyRB8SwWJQfZLPIUn1tlgCOp7BMGz9S/Gx mJtZ7EoZ3MI+plLxf/8/zURDdBxkSGhB0fihKOTYS5/HA6aAGj6xhFDFbFZMR0QRamxLC1e07QU8LwZb7mHVdKu17zLWv3hq tq4LToqoVN0hi6Qh65RA92jJmohiR6Qa/ozXl23p0P53M+uYUOydoAc7XL+rvmqg=</latexit> Y <latexit sha1_base64="wVsefuj4A71dBp3jPzQ+93dXKxs= ">ACB3icbVDLSgMxFM3UV62vqks3wSK4KjNV0GXRjcsK9iHtUDLpnTY0kwxJRihDP8APcKuf4E7c+hl+gb9hp2FbT0QOJxzL/ fkBDFn2rjut1NYW9/Y3Cpul3Z29/YPyodHLS0TRaFJZeqExANnAloGmY4dGIFJAo4tIPxbea3n0BpJsWDmcTgR2QoWMgoMVbq9i JiRpTw9HaL1fcqjsDXiVeTioR6Nf/ukNJE0iEIZyonXc2Pjp0QZRjlMS71EQ0zomAyha6kgEWg/nUWe4jOrDHAolX3C4Jn6dy MlkdaTKLCTWUS97GXif143MeG1nzIRJwYEnR8KE46NxNn/8YApoIZPLCFUMZsV0xFRhBrb0sIVbXsBTxrxlvuYZW0alXvolq7v 6zUb/KOiugEnaJz5KErVEd3qIGaiCKJXtArenOenXfnw/mcjxacfOcYLcD5+gXsiZqp</latexit> GY 2 <latexit sha1_base64="6VoUEA0hP01E3j1j8xL/LIcIFc= ">ACGHicbVDLSsNAFJ34rPUVdaebwSK4KkVdFlwocsK9iFtLJPpTt08mBmIpQ8Dv8ALf6Ce7ErTu/wN9wkmZhWw8MnHvOvd w7x404k8qyvo2l5ZXVtfXSRnlza3tn19zb8kwFhSaNOSh6LhEAmcBNBVTHDqRAOK7HNru+Crz248gJAuDOzWJwPHJMGAeo0RpqW 8e9nyiRq6XKf9pJY+5CUlPLlP+2bFqlo58CKxC1JBRp986c3CGnsQ6AoJ1J2bStSTkKEYpRDWu7FEiJCx2QIXU0D4oN0kvwPKT 7RygB7odAvUDhX/04kxJdy4ru6MztRznuZ+J/XjZV36SQsiGIFAZ0u8mKOVYizQPCACaCKTzQhVDB9K6YjIghVOraZLVLnAgJ4l ow9n8MiadWq9lm1dnteqVtFRiV0hI7RKbLRBaqjG9RATUTRE3pBr+jNeDbejQ/jc9q6ZBQzB2gGxtcvfm2hUg=</latexit> Figure 1: We interpret a translation language model p(Yj|X, Y−j) (Y−j means that phrase Yj has been re- moved from sequence Y) as a causal model. The ran- domness of the causal model comes from unobserved variables G. For data augmentation, we sample coun- terfactual parallel sequences based on the causal effects singled-out by an unsupervised alignment model (i.e., the …
Content truncated to 500 words
Labeled Text
AbstractWe propose a data augmentation method for neural machine translation. It works by interpreting language models and phrasal alignment causally. Specifically, it creates augmented parallel translation corpora by generating (path-specific) counterfactual aligned phrases. We generate these by sampling new source phrases from a masked language model, then sampling an aligned counterfactual target phrase by noting that a translation language model can be interpreted as a Gumbel-Max Structural Causal Model (Oberst and Sontag, 2019). Compared to previous work, our method takes both context and alignment into account to maintain the symmetry between source and target sequences. Experiments on IWSLT’15 English → Vietnamese, WMT’17 English → German, WMT’18 English → Turkish, and WMT’19 robust English → French show that the method can improve the performance of translation, backtranslation and translation robustness.
Publication Details
- Paper ID:
- 20951
- Year:
- 2021
- Publication:
- TIPSTER
- Publication Type:
- conf
- DOI:
- nan