Investigating Terminology Translation in Statistical and Neural Machine Translation: A Case Study on English-to-Hindi and Hindi-to-English

Proceedings of Recent Advances in Natural Language Processing, pages 437–446, Varna, Bulgaria, Sep 2–4, 2019. https://doi.org/10.26615/978-954-452-056-4_052 437 Investigating Terminology Translation in Statistical and Neural Machine Translation: A Case Study on English-to-Hindi and Hindi-to-English Rejwanul Haque,† Mohammed Hasanuzzaman‡ and Andy Way† ADAPT Centre †School of Computing, Dublin City University, Dublin, Ireland ‡Department of Computer Science, Cork Institute of Technology, Cork, Ireland firstname.lastname@adaptcentre.ie Abstract Terminology translation plays a critical role in domain-specific machine trans- lation (MT). In this paper, we con- duct a comparative qualitative evalua- tion on terminology translation in phrase- based statistical MT (PB-SMT) and neu- ral MT (NMT) in two translation di- rections: English-to-Hindi and Hindi-to- English. For this, we select a test set from a legal domain corpus and create a gold standard for evaluating terminol- ogy translation in MT. We also propose an error typology taking the terminology translation errors into consideration. We evaluate the MT systems’ performance on terminology translation, and demonstrate our findings, unraveling strengths, weak- nesses, and similarities of PB-SMT and NMT in the area of term translation. 1 Introduction Over the last five years, there has been incremen- tal progress in the field of NMT (Bahdanau et al., 2015; Vaswani et al., 2017) to the point where some researchers are claiming parity with human translation (Hassan et al., 2018). Nowadays, NMT is regarded as a preferred alternative to PB-SMT (Koehn et al., 2003) and represents a new state- of-the-art in MT research. The rise of NMT has resulted in a swathe of research in the field of MT, unraveling the strengths, weaknesses, impacts and commercialisation aspects of the classical (i.e. PB-SMT) and emerging (i.e. NMT) methods (e.g. (Bentivogli et al., 2016; Toral and Way, 2018)). In brief, the NMT systems are often able to produce better translations than the PB-SMT systems. In- terestingly, terminology translation, a crucial fac- tor in industrial translation workflows (TWs), is one of the less explored areas in MT research. In this context, a few studies (Burchardt et al., 2017; Macketanz et al., 2017; Specia et al., 2017), with their focus on high-level evaluation, have in- dicated that NMT lacks effectiveness in translat- ing domain terms compared to PB-SMT. In this work, we aim to compare PB-SMT and NMT in relation to terminology translation, by carrying out a thorough manual evaluation. For this, we se- lect a test set from legal domain data (i.e. judicial proceedings), and create a gold standard evalua- tion test set following a semi-automatic terminol- ogy annotation strategy. We inspected the patterns of the term translation-related errors in MT. From our observations we make a high-level classifica- tion of the terminology translation-related errors and propose an error typology. We discuss various aspects of terminology translation in MT consid- ering each of the types from the proposed termi- nology translation typology, and dig into the ex- tent of the term translation problems in PB-SMT and NMT with statistical measures as well as lin- guistic analysis. For experimentation, we select a less examined and low-resource language pair, …

Content truncated to 500 words