Using machine learning for system-internal evaluation of transferred linguistic representations

Using Machine Learning for System-Internal Evaluation of Transferred Linguistic Representations Michael Gamon, Hisami Suzuki, and Simon Corston-Oliver Microsoft Research One Microsoft Way Redmond, WA 98052 USA {mgamon, hisamis, simonco}@microsoft.com Abstract We present an automated, system-internal evaluation technique for linguistic representations in a large-scale, multilingual MT system. We use machine-learned classifiers to recognize the differences between linguistic representations generated from transfer in an MT context from representations that are produced by "native" analysis of the target language. In the MT scenario, convergence of the two is the desired result. Holding the feature set and the learning algorithm constant, the accuracy of the classifiers provides a measure of the overall difference between the two sets of linguistic representations: classifiers with higher accuracy correspond to more pronounced differences between representations. More importantly, the classifiers yield the basis for error-analysis by providing a ranking of the importance of linguistic features. The more salient a linguistic criterion is in discriminating transferred representations from "native" representations, the more work will be needed in order to get closer to the goal of producing native-like MT. We present results from using this approach on the Microsoft MT system and discuss its advantages and possible extensions. Keywords Machine translation, evaluation, machine learning, logical form, decision tree. 1. Introduction The evaluation of MT systems falls into two broad categories: cross-system evaluation, and system-internal evaluation. Cross-system evaluations tend to be performed infrequently, and require system-independent evaluation metrics. System-internal evaluation, on the other hand, is customized for a particular MT architecture and needs to be performed on a regular basis, in order to provide feedback to system developers and linguists. Given the high cost in time and money that is associated with human evaluation, automating the evaluation process is crucial in system-internal evaluation (for related efforts, see also Corston-Oliver et al. (2001), Ringger et al. (2001), Bangalore et al (2000), Alshawi et al. (1998) and Su et al. (1992)). Ideally, an automated evaluation procedure should provide two kinds of information: raw numbers that can be used for quantitative analysis over time, and information that helps in qualitative error analysis. In this paper, we propose an automated system-internal evaluation procedure for transferred semantic representations that fits these desiderata. We present results from using this evaluation procedure on the multilingual Microsoft MT system, and show how this approach can be used for error analysis. In an MT system that transfers linguistic representations from a source language to a target language, it is important to ensure that the transferred linguistic representations are as similar as possible to the representations of the target language. In the Microsoft Natural Language Processing System (Heidorn 2000) we use logical form representations (LFs) in semantic transfer. These representations are graphs, representing the predicate argument structure and major semantic relations within a sentence. The nodes in the graph are identified by the lemma of a content word. The edges are directed, labelled arcs, indicating the semantic relationship between nodes. An example of an LF graph is given in Figure 1. Note …

Content truncated to 500 words