Accenting unknown words in a specialized language

Accenting unknown words in a specialized language Pierre Zweigenbaum and Natalia Grabar DIAM — STIM/DSI, Assistance Publique – Hôpitaux de Paris & Département de Biomathématiques, Université Paris 6 {ngr,pz}@biomath.jussieu.fr Abstract We propose two internal methods for ac- centing unknown words, which both learn on a reference set of accented words the contexts of occurrence of the various ac- cented forms of a given letter. One method is adapted from POS tagging, the other is based on finite state transducers. We show experimental results for letter e on the French version of the Medical Subject Headings thesaurus. With the best training set, the tagging method ob- tains a precision-recall breakeven point of 84.2 �4.4% and the transducer method 83.8�4.5% (with a baseline at 64%) for the unknown words that contain this let- ter. A consensus combination of both in- creases precision to 92.0�3.7% with a re- call of 75%. We perform an error analysis and discuss further steps that might help improve over the current performance. 1 Introduction The ISO-latin family, Unicode or the Universal Character Set have been around for some time now. They cater, among other things, for letters which can bear different diacritic marks. For instance, French uses four accented es (éèêë) besides the unaccented form e. Some of these accented forms correspond to phonemic differences. The correct handling of such accented letters, beyond US ASCII, has not been immediate and general. Although suitable charac- ter encodings are widely available and used, some texts or terminologies are still, for historical rea- sons, written with unaccented letters. For instance, in the French version of the US National Library of Medicine’s Medical Subject Headings thesaurus (MeSH, (INS, 2000)), all the terms are written in unaccented uppercase letters. This causes difficul- ties when these terms are used in Natural Language interfaces or for automatically indexing textual doc- uments: a given unaccented word may match several words, giving rise to spurious ambiguities such as, e.g., marche matching both the unaccented marche (walking) and the accented marché (market). Removing all diacritics would simplify match- ing, but would increase ambiguity, which is al- ready pervasive enough in natural language pro- cessing systems. Another of our aims, besides, is to build language resources (lexicons, morpho- logical knowledge bases, etc.) for the medi- cal domain (Zweigenbaum, 2001) and to learn lin- guistic knowledge from terminologies and cor- pora (Grabar and Zweigenbaum, 2000), including the MeSH. We would rather work, then, with lin- guistically sound data in the first place. We therefore endeavored to produce an accented version of the French MeSH. This thesaurus in- cludes 19,971 terms and 9,151 synonyms, with 21,475 different word forms. Human reaccentua- tion of the full thesaurus is a time-consuming, error- prone task. As in other instances of preparation of linguistic resources, e.g., part-of-speech-tagged cor- pora or treebanks, it is generally more efficient for a human to correct a first annotation than to produce it from scratch. This can also help obtain better con- sistency over volumes of data. The issue …

Content truncated to 500 words