The Learnability of the Annotated Input in NMT Replicating (Vanmassenhove and Way, 2018) with OpenNMT

Proceedings of the 12th Conference on Language Resources and Evaluation (LREC 2020), pages 5631–5640 Marseille, 11–16 May 2020 c⃝ European Language Resources Association (ELRA), licensed under CC-BY-NC 5631 The Learnability of the Annotated Input in NMT Replicating (Vanmassenhove and Way, 2018) with OpenNMT Nicolas Ballier1, Nabil Amari2, Laure Merat3, Jean-Baptiste Yun`es4 Universit´e de Paris1,2,3,4 CLILLAC-ARP, F-75013 Paris, France1, UFR d’informatique2,3 & IRIF4, F-75013 Paris, France {nicolas.ballier, jean-baptiste.yunes}@u-paris.fr, {nabil.amari,laure.merat}@etu.univ-paris-diderot.fr Abstract In this paper, we reproduce some of the experiments related to neural network training for Machine Translation as reported in (Vanmassenhove and Way, 2018). They annotated a sample from the EN-FR and EN-DE Europarl aligned corpora with syntactic and semantic annotations to train neural networks with the Nematus Neural Machine Translation (NMT) toolkit. Following the original publication, we obtained lower BLEU scores than the authors of the original paper, but on a more limited set of annotations. In the second half of the paper, we try to analyze the difference in the results obtained and suggest some methods to improve the results. We discuss the Byte Pair Encoding (BPE) used in the pre-processing phase and suggest feature ablation in relation to the granularity of syntactic and semantic annotations. The learnability of the annotated input is discussed in relation to existing resources for the target languages. We also discuss the feature representation likely to have been adopted for combining features. Keywords: NMT, replication study, learnability, corpus annotation 1. Introduction To validate a scientific experiment, its reproducibility is one of the bases of the scientific process because it may be proven wrong (Popper, 2005). As is well-known, if a repeated experiment gives different results, its validity is questioned as it fails to be generalised. The preoccupation for reproducibility when using Neural Networks (NN) is a frequent issue raised in their different applications. For ex- ample, (Laje and Buonomano, 2013) have analysed robust- ness against noise in Recurrent Neural Networks (RNN) used to investigate the stability of complex spatiotemporal motor patterns. For neural networks used in Neural Machine Translation, reproducibility seems even more problematic as NN are fed with variable samples of the training data. To this end, the translation toolkit OpenNMT (Klein et al., 2017) in its Py- Torch implementation, now has access to pseudorandom number generators, which offer better control on experi- ments, even though “completely reproducible results are not guaranteed across PyTorch releases”.1 We have used the version OpenNMT-py v1.0.0.rc1 (released on 1st of Oct 2019) of OpenNMT (PyTorch 1.2) to try to reproduce the experiments described in (Vanmassenhove and Way, 2018) as part of an on-going research project on the possibility to improve quality translation in NMT with linguistically an- notated input in the training phase. The remainder of the paper is organised as follows2: sec- tion 2 presents the method of the original paper (Van- massenhove and Way, 2018). Section 3 delineates our repli- cation process. Section 4 details how our experiments may further the original analysis. Section 5 explains our direc- 1https://pytorch.org/docs/stable/notes/ randomness.html 2We have been inspired by …

Content truncated to 500 words