Candidate Ranking for Maintenance of an Online Dictionary

Candidate Ranking for Maintenance of an Online Dictionary Claire Broad1,2, Helen Langone1, David Guy Brizan2 1Dictionary.com 555 12th Street, Oakland, CA 94607 2University of San Francisco 101 Howard Street, San Francisco, CA 94105 {Claire.Broad, Helen.Langone}@dictionary.com, dgbrizan@usfca.edu Abstract Traditionally, the process whereby a lexicographer identifies a lexical item to add to a dictionary – a database of lexical items – has been time-consuming and subjective. In the modern age of online dictionaries, all queries for lexical entries not currently in the database are indistinguishable from a larger list of misspellings, meaning that potential new or trending entries can get lost easily. In this project, we develop a system that uses machine learning techniques to assign these “misspells” a probability of being a novel or missing entry, incorporating signals from orthography, usage by trusted online sources, and dictionary query patterns. Keywords: Neologisms, machine-readable dictionary, ranking 1. Introduction Dictionaries are databases in which the primary entities are words. Like a (non-temporal) database, a dictionary’s con- tents are frozen in time (Guthrie et al., 1996; Labov, 2011; Curzan, 2012). Therefore, for a dictionary to remain rel- evant, new lexical entries for entirely new words – neol- ogisms – must be added. This maintenance is important for machine-readable dictionaries (MRDs) as well as those built for human consumption. We have created and continue to maintain a descriptive, general-purpose dictionary of American English1. Through our publicly-available web site, this lexical database is searchable by any of our users. Each lexical item may in- clude one or more spellings, parts of speech, definitions, pronunciations, origins, examples of usage, and other in- formation. Each month, our site hosts approximately 70 million users who collectively generate more than 450 mil- lion searches. The content of our site is maintained by lexicographers. Some of this maintenance involves researching candidate lexical entries drawn from a number of sources, including unmatched queries: users’ searches on our site which fail to match an item in the database. Given the size of our database and the number of unmatched queries, the work of prioritizing the items to be considered for inclusion is labor-intensive and somewhat subjective. Figure 1 contains an overview of our procedure for maintaining our dictio- nary. In the interest of establishing a reasonable scope for this project, we limit our focus to single-word items. However, we believe that the same process could be effectively ap- plied to multiword queries with some adjustment for the specific lexical considerations of phrases. The goal of this effort is the production of a ranking for the unmatched query list to help our lexicographers iden- tify potential candidate entries and focus attention on the 1Many non-American English lexemes (eg. “colour”) are also included in our dictionary as variant spellings of their American counterparts. Figure 1: Overview of Maintenance Procedure. items most qualified to be included in the lexicon. This work is made difficult because our candidates have no con- textual information which has proven useful in the past (Weischedel et al., 1993; Nakagawa et al., 2001). …

Content truncated to 500 words