Discriminative Language Models as a Tool for Machine Translation Error Analysis

Proceedings of COLING 2014, the 25th International Conference on Computational Linguistics: Technical Papers, pages 1124–1132, Dublin, Ireland, August 23-29 2014. Discriminative Language Models as a Tool for Machine Translation Error Analysis Koichi Akabe Graham Neubig Sakriani Sakti Tomoki Toda Satoshi Nakamura Graduate School of Information Science Nara Institute of Science and Technology 8916-5 Takayama-cho, Ikoma-shi, Nara, Japan {akabe.koichi.zx8, neubig, ssakti, tomoki, s-nakamura}@is.naist.jp Abstract In this paper, we propose a new method for effective error analysis of machine translation (MT) systems. In previous work on error analysis of MT, error trends are often shown by frequency. However, if we attempt to perform a more detailed analysis based on frequently erroneous word strings, the word strings also often occur in correct translations, and analyzing these correct sen- tences decreases the overall efficiency of error analysis. In this paper, we propose the use of regularized discriminative language models (LMs) to allow for more focused MT error analysis. In experiments, we demonstrate that our method is more efficient than frequency-based analysis, and examine differences across systems, language pairs, and evaluation measures. 1 1 Introduction Accuracy of Statistical Machine Translation (SMT) systems is continually increasing, but systems are now more complex than ever before. As a result, not all effects of making modifications to a system are known without actually making the modification and generating translations. Therefore, in the process of developing an SMT system, it is common to evaluate actual translations to identify problems to make improvements. This process is time consuming, as it is often necessary to analyze a large number of translations to get an overall grasp of the system’s error trends. In addition, many sentences will contain no errors, or only errors from the long tail that are not representative of the system as a whole. On the other hand, if we are able to detect and rank important errors automatically, we will likely be able to find representative errors of the SMT system more efficiently. Previous work has proposed methods for automatic error analysis of MT systems based on automati- cally separating errors into classes and sorting these classes by frequency (Vilar et al., 2006; Popovic and Ney, 2011). These classes cover common mistakes of MT systems, e.g. conjugation, reordering, word deletion, and insertion. This makes it possible to view overall error trends, but when the goal of analysis is to identify errors to make some concrete improvement to the system, it is often necessary to perform a more focused analysis, looking at actual errors made by a particular language pair or system. We show examples of errors types that are informative, but are language- or task-specific, and not covered by pre- vious methods in Figure 1. In this example, the type given by more standard error typologies is indicated by “Traditional type,” but we would prefer a more detailed analysis such as “Fine-grained type,” would allows us to take specific steps to fix the machine translation system (such as ensuring that Wikipedia titles are not punctuated, or normalizing full-width …

Content truncated to 500 words