MTLens: Machine Translation Output Debugging

Proceedings of the 13th Conference on Language Resources and Evaluation (LREC 2022), pages 4221–4226 Marseille, 20-25 June 2022 © European Language Resources Association (ELRA), licensed under CC-BY-NC-4.0 4221 MTLens: Debugging Machine Translation Systems Based on Their Output Shreyas Sharma, Kareem Darwish, Lucas Aguiar Pavanelli, Thiago Castro Ferreira Mohamed Al-Badrashiny, Kamer Ali Yuksel, Hassan Sawaf aiXplain Inc. California, USA {shreyas.sharma, kareem.darwish, lucas.pavanelli, thiago, mohamed, kamer, hassan}@aixplain.com Abstract The performance of Machine Translation (MT) systems varies significantly with inputs of diverging features such as topics, genres, and surface properties. Though there are many MT evaluation metrics that generally correlate with human judgments, they are not directly useful in identifying specific shortcomings of MT systems. In this demo, we present a benchmarking inter- face that enables improved evaluation of specific MT systems in isolation or multiple MT systems collectively by quantitatively evaluating their performance on many tasks across multiple domains and evaluation metrics. Further, it facilitates effective debugging and error analysis of MT output via the use of dynamic filters that help users hone in on problem sentences with specific properties, such as genre, topic, sentence length, etc. The interface can be extended to include additional filters such as lexical, morphological, and syntactic features. Aside from helping debug MT output, it can also help in identifying problems in reference translations and evaluation metrics. Keywords: Machine Translation, Evaluation, Error Analysis 1. Introduction The performance of Machine Translation (MT) sys- tems may vary significantly across genres, topics, sen- tence surface properties (ex. length, punctuation, POS tags, etc.), and styles. Though much work has been done on MT evaluation in terms of metrics and datasets, performing error analysis efficiently is cumbersome and time-consuming. In this demo, we present an in- tuitive MT benchmarking interface for quantitatively evaluating and debugging MT systems together or in isolation along different properties of the text, such as genre and topic, and using a variety of MT evaluation metrics and evaluation test sets. Our proposed system helps researchers and practitioners identify problem ar- eas for their MT model and whether these problems are specific to their models or shared by other mod- els. Such can greatly simplify error analysis and help guide further research and development efforts. Addi- tionally, our system has a modular design that can be extended to handle additional metrics and test set fea- tures. Though not directly shown in the demo, users can choose the test sets, metrics, and models of their choice, and the system would perform benchmarking per their preferences. When the benchmark results are ready and stored in a database, the user can in- teract with the evaluation results and perform system debugging. In the online demo (https://bit.ly/ 3IdHzBv), we show the capability of the system by evaluating 6 commercial MT systems1 on 2 language pairs using 4 different test sets from OPUS2 and 5 dif- ferent evaluation metrics. The contributions of this demo are as follows: 1For legal reasons, the system names are anonymized. 2https://opus.nlpl.eu/ • We present an intuitive user interface that …

Content truncated to 500 words