Measuring MT Adequacy Using Latent Semantic Analysis

Measuring MT Adequacy Using Latent Semantic Analysis Florence Reeder MITRE Corporation 7515 Colshire Dr. McLean VA 22102 freeder@mitre.org Abstract Translation adequacy is defined as the amount of semantic content from the source language document that is con- veyed in the target language document. As such, it is more difficult to measure than intelligibility since semantic content must be measured in two documents and then compared. Latent Semantic Analysis is a content measurement technique used in language learner evaluation that exhib- its characteristics attractive for re-use in machine translation evaluation (MTE). This experiment, which is a series of ap- plications of the LSA algorithm in various configurations, demonstrates its useful- ness as an MTE metric for adequacy. In addition, this experiment lays the groundwork for using LSA as a method to measure the accuracy of a translation without reliance on reference translations. 1 Introduction Translation adequacy can be defined as the amount of semantic content from the source lan- guage document that is conveyed in the target lan- guage document (White & Reeder, 2002). As such, it is difficult to measure since semantic con- tent must be measured in two documents and then compared. Latent Semantic Analysis (Furnas et al., 1988; Deerwester et al., 1990; Landauer, et al., 1998a; Foltz et al., 1998; Foltz et al., 2000) is a content measurement technique used in language learner evaluation that exhibits characteristics at- tractive for re-use in machine translation evalua- tion (MTE). In particular, LSA measures semantic content, demonstrates independence over individ- ual word choice, tolerance of syntactic errors, abil- ity to train for domains and applicability to language testing problems. For instance, it has been used for assessing the Test of English as a Foreign Language (TOEFL) essays (Burstein et al., 1998a; Burstein & Chodorow, 1999; Landauer et al., 1998a). This experiment, which is a series of applications of the LSA algorithm in various con- figurations, demonstrates its usefulness as an MTE metric for adequacy. In addition, this experiment lays the groundwork for using LSA as a method to measure the accuracy of a translation without reli- ance on reference translations. 2 Latent Semantic Analysis Latent Semantic Analysis (LSA) was developed from the information retrieval technique, Latent Semantic Indexing (LSI) (Furnas et al., 1988; Deerwester et al., 1990). LSA has been success- fully applied to the problem of automated essay grading (Foltz, 1996; Foltz et al., 1996; Foltz et al., 1998; Foltz et al., 1999; Foltz et al., 2000; Laham, 1997; Landauer et al., 1998a, 1998b; Rehder, et al., 1998; Landauer & Psotka, 2000; Burstein et al., 1998a; Burstein & Chodorow, 1999); intelligent tutoring systems (Foltz et al., 2000; Wiemer- Hastings, et al., 1999; Graesser et al., 2000a) and other educational problems (Landauer & Psotka, 2000). The basic concept behind LSA is that words are known by the company they keep, or co- occurrence, and also by the functions they serve, or usage patterns. The developers also assume word dependence unlike other information retrieval techniques (Deerwester et al., 1990). LSA at- tempts …

Content truncated to 500 words