NMT and PBSMT Error Analyses in English to Brazilian Portuguese Automatic Translations

Proceedings of the 12th Conference on Language Resources and Evaluation (LREC 2020), pages 3623–3629 Marseille, 11–16 May 2020 c⃝ European Language Resources Association (ELRA), licensed under CC-BY-NC 3623 NMT and PBSMT Error Analyses in English to Brazilian Portuguese Automatic Translations Helena de Medeiros Caseli, Marcio Lima In´acio Federal University of S˜ao Carlos, University of S˜ao Paulo Via Washington Lu´ıs, km 235, CP 676, CEP 13565-905, S˜ao Carlos, SP, Brazil helenacaseli@ufscar.br, marciolimainacio@usp.br Abstract Machine Translation (MT) is one of the most important natural language processing applications. Independently of the applied MT approach, a MT system automatically generates an equivalent version (in some target language) of an input sentence (in some source language). Recently, a new MT approach has been proposed: neural machine translation (NMT). NMT systems have already outperformed traditional phrase-based statistical machine translation (PBSMT) systems for some pairs of languages. However, any MT approach outputs errors. In this work we present a comparative study of MT errors generated by a NMT system and a PBSMT system trained on the same English – Brazilian Portuguese parallel corpus. This is the first study of this kind involving NMT for Brazilian Portuguese. Furthermore, the analyses and conclusions presented here point out the specific problems of NMT outputs in relation to PBSMT ones and also give lots of insights into how to implement automatic post-editing for a NMT system. Finally, the corpora annotated with MT errors generated by both PBSMT and NMT systems are also available. Keywords: Machine Translation, Corpus, Evaluation Methodologies 1. Introduction Machine Translation (MT) is one of the most important Natural Language Processing (NLP) applications. In the last 70 years, many different approaches have been pro- posed for MT such as the rule-based MT (Senellart and Senellart, 2005; Armentano-Oller et al., 2006), the statis- tical MT (Brown et al., 1993; Och and Ney, 2004; Koehn et al., 2007) and, recently, the neural MT (Bahdanau et al., 2015; Klein et al., 2017). Regardless of the MT approach applied, a MT system automatically generates an equiva- lent version (in some target language) of an input sentence (in some source language). However, despite the huge effort of the MT community, it is not possible yet to generate a perfect completely auto- matic translation for unrestricted domains. Regarding the Brazilian Portuguese, this statement has been confirmed by some previous works such as (Caseli, 2007) and (Martins and Caseli, 2015). In (Caseli, 2007), the best MT systems for Brazilian Portuguese in that time1 translated incorrectly more than 50% of the input sentences. Lately, in (Martins and Caseli, 2015), the same occurred for a Phrase-based Machine Translation System (PBSMT), for which 67% of the translated sentences in Brazilian Portuguese had one or more translation errors. More recently, some analyses have also been done for Neural Machine Translation (NMT) models, such as (Ben- tivogli et al., 2016; Klubiˇcka et al., 2017; Popovi´c, 2018). However, to the best of our knowledge, no comparative analysis of MT errors output by PBSMT and NMT has been done for the English and …

Content truncated to 500 words