On the Difficulty of Translating Free-Order Case-Marking Languages

On the Difficulty of Translating Free-Order Case-Marking Languages Arianna Bisazza Ahmet ¨Ust¨un Stephan Sportel Center for Language and Cognition University of Groningen, The Netherlands {a.bisazza, a.ustun}@rug.nl, research@spor.tel Abstract Identifying factors that make certain languages harder to model than others is essential to reach language equality in future Natural Language Processing technologies. Free-order case-marking languages, such as Russian, Latin, or Tamil, have proved more challenging than fixed-order languages for the tasks of syntactic parsing and subject-verb agreement prediction. In this work, we investigate whether this class of lan- guages is also more difficult to translate by state-of-the-art Neural Machine Translation (NMT) models. Using a variety of synthetic languages and a newly introduced translation challenge set, we find that word order flexi- bility in the source language only leads to a very small loss of NMT quality, even though the core verb arguments become impossible to disambiguate in sentences without seman- tic cues. The latter issue is indeed solved by the addition of case marking. However, in medium- and low-resource settings, the overall NMT quality of fixed-order languages remains unmatched. 1 Introduction Despite the tremendous advances achieved in less than a decade, Natural Language Processing re- mains a field where language equality is far from being reached (Joshi et al., 2020). In the field of Machine Translation, modern neural models have attained remarkable quality for high-resource language pairs like German-English, Chinese- English, or English-Czech, with a number of stud- ies even claiming human parity (Hassan et al., 2018; Bojar et al., 2018; Barrault et al., 2019; Popel et al., 2020). These results may lead to the unfounded belief that Neural Machine Transla- tion (NMT) methods will perform equally well in any language pair, provided similar amounts of training data. In fact, several studies suggest the opposite (Platanios et al., 2018; Ataman and Federico, 2018; Bugliarello et al., 2020). Why, then, do some language pairs have lower translation accuracy? And, more specifically: Are certain typological profiles more challenging for current state-of-the-art NMT models? Every lan- guage has its own combination of typological properties, including word order, morphosyntac- tic features, and more (Dryer and Haspelmath, 2013). Identifying language properties (or combi- nations thereof) that pose major problems to the current modeling paradigms is essential to reach language equality in future MT (and other NLP) technologies (Joshi et al., 2020), in a way that is orthogonal to data collection efforts. Among oth- ers, natural languages adopt different mechanisms to disambiguate the role of their constituents: Flexible order typically correlates with the pres- ence of case marking and, vice versa, fixed order is observed in languages with little or no case marking (Comrie, 1981; Sinnem¨aki, 2008; Futrell et al., 2015b). Morphologically rich languages in general are known to be challenging for MT at least since the times of phrase-based statisti- cal MT (Birch et al., 2008) due to their larger and sparser vocabularies, and remain challenging even for modern neural architectures (Ataman and Federico, 2018; Belinkov et al., 2017). By con- trast, the relation between word …

Content truncated to 500 words