Measuring Semantic Entropy

m m m [] m mm m m m m m m m n m mm I mm m m m Measuring Semantic Entropy Dept. I. Dan Melamed of Computer and Information Science University of Pennsylvania Philadelphia, PA, 19104, U.S.A. melamed@unagi, cis. upenn, edu Abstract Semantic entropy is a measure of se- mantic -mbiguity and uninformative- hess. It is a graded lexical feature which may play a role anywhere lex- ical semantics plays a role. This pa- per presents a method for measuring semantic entropy using translational distributions of words in parallel text corpora. The measurement method is well-defined for all words, including function words, and even for punctu- ation. 1 Introduction Semantic entropy is a measure of semantic ambi- gnity and uninformativeness. This paper presents a method for measuring semantic entropy using trans- lational distributions of words in parallel text cor- pora. The measurement method is well-defined for all words, including function words, and even for punctuation. The hypothesis behind the measure- ment method is that semantically heavy words are more likely to have unique counterparts in other lan- guages, so they tend to be translated more consis- tently than semantically lighter words. The consis- tency with which words are translated can be calcu- lated from the translational distributions of words in parallel texts in two languages (bitexts). The translational distributions can be estimated using any reasonably good word translation model, such as those described in (BD+93; Che96) or in (Me196b). Semantic entropy is a graded lexical feature which may play a role anywhere lexical semantics plays a role. For example, semantic entropy can be inter- preted as semantic ambiguity. On this interpreta- tion, it can predict the difficulty of disambiguating the sense of a given word. Brown et al. (BD+91) present a word-sense disambiguation algorithm in- volving minimization of semantic entropy weighted by word frequency. Yarowsky (Yar93) compares the semantic entropy of homographs conditioned on dif- ferent contexts. Another way to use semantic en- tropy for word-sense disambiguation is to allow dis- ambiguation algorithms that favor precision over re- call to ignore words with high semantic entropy. In the same vein, developers of interlinguas for machine translation can use semantic entropy to predict the required complexity of lexical elements of the repre- sentation. Another interpretation of entropy is as the inverse of reliability. Machine learning algorithms may ben- efit from discounting the importance of data that has high entropy. For example, an algorithm learning se- lectional preferences may not want to generalize the statistical characteristics of "take into account" to other objects of "take," if it knows that "take" has high semantic entropy. I.e. the selectional prefer- ences of "take" are hard to predict because it usu- ally functions as a support verb. Resnik has used semantic entropy to explore selectional preferences, although he measured it in a different way (Res93). Semantic entropy can help researchers decide not only how to work with words, but also which words to work with. Several applications in computational linguistics use stop-lists …

Content truncated to 500 words