Identifying main obstacles for statistical machine translation of morphologically rich South Slavic languages

Poor man’s lemmatisation for automatic error classification Maja Popovi´c1 Mihael Arˇcan2 Eleftherios Avramidis1 Aljoscha Burchardt1 Arle Lommel1 1 DFKI – Language Technology Lab, Berlin, Germany firstname.lastname@dfki.de 2 Insight Centre for Data Analytics, National University of Galway, Ireland mihael.arcan@insight-centre.org Abstract This paper demonstrates the possibility to make an existing automatic error classi- fier for machine translations independent from the requirement of lemmatisation. This makes it usable also for smaller and under-resourced languages and in situa- tions where there is no lemmatiser at hand. It is shown that cutting all words into the first four letters is the best method even for highly inflective languages, preserving both the detected distribution of error types within a translation output as well as over various translation outputs. The main cost of not using a lemmatiser is the lower accuracy of detecting the in- flectional error class due to its confusion with mistranslations. For shorter words, actual inflectional errors will be tagged as mistranslations, for longer words the other way round. Keeping all that in mind, it is possible to use the error classifier without target language lemmatisation and to ex- trapolate inflectional and lexical error rates according to the average word length in the analysed text. 1 Introduction Future improvement of machine translation (MT) systems requires reliable automatic evaluation and error classification tools in order to minimise ef- forts of time and money consuming human clas- sification. Therefore automatic error classification tools have been developed in recent years (Zeman c⃝ 2015 The authors. This article is licensed under a Creative Commons 3.0 licence, no derivative works, attribution, CC- BY-ND. et al., 2011; Popovi´c, 2011) and are being used to facilitate the error analysis. Although these tools are completely language independent, for obtain- ing a precise error distribution over classes a lem- matiser for the target language is required. For the languages strongly supported in language re- sources and tools this does not pose a problem. However, for a number of languages a lemmatiser might not be at hand, or it does not exist at all. This paper investigates possibilities for obtaining reasonable error classification results without lem- matisation. To the best of our knowledge, this issue has not been investigated so far. 2 Motivation and explored methods We investigate the edit-distance i.e. word error rate (WER) approach implemented in the Hjer- son tool (Popovi´c, 2011), which enables detec- tion of five error categories: inflectional errors, word order errors, missing words (omissions), ex- tra words (additions) and lexical errors (mistrans- lations). For a given MT output and reference translation, the classication results are provided in the form of the five error rates, whereby the num- ber of errors for each category is normalised over the total number of words. The detailed description of the approach can be found in (Popovi´c and Ney, 2011). The starting point is to identify actual words contributing to the Word Error Rate (WER), recall (reference) error rate (RPER) and precision (hypothesis) error rate (HPER). The WER errors are marked as substitu- …

Content truncated to 500 words