A Quantitative Analysis of Word-Definition in a Machine-Readable Dictionary

A Quantitative Analysis of Word-Definition in a Machine-Readable Dictionary Robert W.P. Luk and Venus M.K. Chan Department of Chinese, Translation and Linguistics, City University of Hong Kong rwpluk©ctspc05.cityu.edu.hk Abstract This paper investigates some of the distributional properties of word definitions in a machine- readable dictionary which • was obtained from the Oxford Text Archive. Three types of dis- tributions were examined: (1) frequency-ranked distribution of words in the definitions, (2) the length distribution of word definitions and (3) the frequency distribution of the number of unique tags of an entry. In addition, the coverage characteristics of headwords over word defi- nitions are also explored. A rough-and-ready comparison of distributional properties between tokens and their morphologically decomposed ones are made. Our result shows that morpho- logical dcomposition does not change the length distribution of the word definitions nor the ranked-frequency distribution of words, significantly. However, it increases the coverage of word definitions dramatically compared with no decompositions. Furthermore, the frequency distribution of the number of unique tags per entry is approximately linear when the data is suitablely scaled (i.e. linear or logarithmic). 1 Introduction Many (English) dictionaries (e.g. Collins [1], OALD [2], Longman [3]) have been made available on- line but they have rarely been used as data in quantitative analysis. Preliminary analysis [4] such as compiling word usage frequency in word definitions, thereby compiling conceptual primitives has been carried out. A notable exception is the mathematical modeling of word length distribution [5]. In this paper, we focus on a quantitative analysis of word definitions. In particular, we examined: 1. the ranked-frequency distribution of words in the definitions; 2. the length distribution of word definitions; 3. the frequency distribution of the number of unique tags in word definitions; 4. the coverage statistics of words in dictionary definitions. These quantitative descriptions facilitate a more objective comparison between different dictionar- ies than merely the number of entries in a dictionary. Ultimately, they can serve as part of a more comprehensive evaluation metrics for dictionaries. For example, word length distribution may indicate whether definitions are given comparatively concisely or brief. Unusual distribution of the number of unique tags might indicate bias or undersampling of the data which might be intended. Poor coverage characteristics might be due to less adherence in using a control vocabulary. Furthermore, these quantitative descriptions can be used to guide the construction of computational lexicon. For example, coverage statistics can be used to determine the size of the control vocabulary in analyzing word definitions as in [6]. Apart from applications, quantitative analysis provides exploratory data for quantitative modeling, giving a more comprehensive description of the phenomenon at hand. The quantitative model of word length distribution [5] is a good example. 133 ( Doh' titt� \I/decocting) Resultbck ) \Lust name in alphabetical order Resulttct Distribution et Numbery d Definition per Tag Distribution of POS )("-- Distribution of the Length et Definitions sort in alphabetical order satin� alphabetical�sort by frequency order I extract� extract words headwords�in defini dors Resuitcvg Morphological Decomposition Distribution of …

Content truncated to 500 words