Robust Open-Vocabulary Translation from Visual Text Representations

Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 7235–7252 November 7–11, 2021. c⃝2021 Association for Computational Linguistics 7235 Robust Open­Vocabulary Translation from Visual Text Representations Elizabeth Salesky1 and David Etter2 and Matt Post1,2 1Center for Language and Speech Processing 2Human Language Technology Center of Excellence Johns Hopkins University Baltimore, Maryland, USA Abstract Machine translation models have discrete vo­ cabularies and commonly use subword seg­ mentation techniques to achieve an ‘open vo­ cabulary.’ This approach relies on consis­ tent and correct underlying unicode sequences, and makes models susceptible to degrada­ tion from common types of noise and vari­ ation. Motivated by the robustness of hu­ man language processing, we propose the use of visual text representations, which dispense with a finite set of text embeddings in favor of continuous vocabularies created by process­ ing visually rendered text with sliding win­ dows. We show that models using visual text representations approach or match per­ formance of traditional text models on small and larger datasets. More importantly, mod­ els with visual embeddings demonstrate sig­ nificant robustness to varied types of noise, achieving e.g., 25.9 BLEU on a character per­ muted German–English task where subword models degrade to 1.9. 1 Introduction Machine translation models degrade quickly in the presence of noise, such as character swaps or misspellings (Belinkov and Bisk, 2018; Khayral­ lah and Koehn, 2018; Eger et al., 2019). Part of the reason for this brittleness is the reliance of MT systems on subword segmentation (Sennrich et al., 2016) as the solution for the open­vocabulary problem, since it can cause even minor variations in text to result in very different token sequences, needlessly fragmenting the data (Table 1). These issues can be mitigated with techniques including normalization, adding synthetic noisy training data (Vaibhav et al., 2019), or often simply moving to larger data settings. However, it is impossible to anticipate all kinds of noise in light of their combinatorics, and in any case, attempts to do so add complexity to the model training process. Phenomena Word BPE (5k) Vowelizationﺏﺎﺘﻛﺏﺎﺘﻛ(1) ُ·ﺏﺍ·ﺕ·ِ·ﻚﻟﺍ(5) Misspelling language language (1) langauge la · ng · au · ge (4) Visually Similar Characters really really (1) rea11y re · a · 1 · 1 · y (5) Shared Character Components 확인한다 확인 · 한 · 다 (3) 확인했다 확인 · 했다 (2) Table 1: Examples of common behavior which cause divergent representations for subword models. Humans, in contrast, are remarkably robust to all kinds of text permutations (Rayner et al., 2006), including extremes such as “l33tspeak” (Perea et al., 2008). It stands to reason that one source of this robustness is that humans process text, not from discrete unicode representations, but visu­ ally, and that modeling this kind of information might yield more human­like robustness. Draw­ ing on this, we propose to translate from visual text representations. Our model still consumes text, but instead of creating an embedding matrix from subword tokens, we render the raw, unsegmented text as images, divide it into overlapping slices, …

Content truncated to 500 words