Reproducing a Morphosyntactic Tagger with a Meta-BiLSTM Model over Context Sensitive Token Encodings

Proceedings of the 12th Conference on Language Resources and Evaluation (LREC 2020), pages 5563–5568 Marseille, 11–16 May 2020 c⃝ European Language Resources Association (ELRA), licensed under CC-BY-NC 5563 Reproducing a Morphosyntactic Tagger with a Meta-BiLSTM Model over Context Sensitive Token Encodings Yung Han Khoe CLS, Radboud University Nijmegen, The Netherlands yhkhoe@protonmail.com Abstract Reproducibility is generally regarded as being a requirement for any form of experimental science. Even so, reproduction of research results is only recently beginning to be practiced and acknowledged. In the context of the REPROLANG 2020 shared task, we contribute to this trend by reproducing the work reported on by Bohnet et al. (2018) on morphosyntactic tagging. Their meta-BiLSTM model achieved state-of-the-art results across a wide range of languages. This was done by integrating sentence-level and single-word context through synchronized training by a meta-model. Our reproduction only partially confirms the main results of the paper in terms of outperforming earlier models. The results of our reproductions improve on earlier models on the morphological tagging task, but not on the part-of-speech tagging task. Furthermore, even where we improve on earlier models, we fail to match the F1-scores reported for the meta-BiLSTM model. Because we chose not to contact the original authors for our reproduction study, the uncertainty about the degree of parallelism that was achieved between the original study and our reproduction limits the value of our findings as an assessment of the reliability of the original results. At the same time, however, it underscores the relevance of our reproduction effort in regard to the reproducibility and interpretability of those findings. The discrepancies between our findings and the original results demonstrate that there is room for improvement in many aspects of reporting regarding the reproducibility of the experiments. In addition, we suggest that different reporting choices could improve the interpretability of the results. Keywords: reproduction, morphosyntactic tagging, meta-bilstm 1. Introduction Determining the state-of-the-art in language related ma- chine learning is not a trivial task. This issue has been recognized for some time (Armstrong et al., 2009; Lops et al., 2011), and has recently been receiving more atten- tion (Dacrema et al., 2019; Lin, 2019). It can be regarded as a symptom of the underlying problem of a research and publication culture that focuses on beating previous bench- marks while disregarding the need to contribute to scientific knowledge and understanding (Sculley et al., 2018). Judging whether real progress has been made in any field of machine learning can be difficult because of two main problems. Firsty, the reported results are not always repro- ducible without investing an unreasonable amount of time and effort. Secondly, the comparisons that are made do not always make sense. This is often because the baseline that is chosen is generally too weak, or because not enough ef- fort is made to optimize the baseline. An inverse relationship between the popularity of a scien- tific method or model, and the reliability of the reporting on its results has been argued for (Ioannidis, 2005) and was empirically …

Content truncated to 500 words