Word Transformation Heuristics Agains Lexicons for Cognate Detection

Word Transformation Heuristics Against Lexicons for Cognate Detection Alexandra L. Uitdenbogerd RMIT University GPO Box 2476, Melbourne VIC 3001 Australia sandrau@rmit.edu.au Abstract One of the most common lexical transfor- mations between cognates in French and English is the presence or absence of a ter- minal “e”. However, many other transfor- mations exist, such as a vowel with a cir- cumflex corresponding to the vowel and the letter s. Our algorithms tested the ef- fectiveness of taking the entire English and French lexicons from Treetagger, deac- centing the French lexicon, and taking the intersection of the two. Words shorter than 6 letters were excluded from the list, and a set of lexical transformations were also used prior to intersecting, to increase the potential pool of cognates. The result was 15% above the baseline cognate list in the initial test set, but only 1% above it in the final test set. However, its accuracy was consistant at about 37% for both test sets. 1 Credits 2 Introduction When assessing readability of English for French native speakers, or French for English native speakers, the cognates — words with similar ap- pearance and meaning — tend to be relatively dif- ficult words, making traditional readability mea- sures less effective than a simple average words per sentence ( Uitdenbogerd, 2005). While most words that look similar in French and English are cognates, some commonly occurring words that look similar, such as “a”, “as”, and “an”, tend to be false friends. Other words are partial cog- nates, having a similar meaning only in some sit- uations (Wang and Sitbon, 2014). Our approach ignored all context and focussed on simple heuris- tics. All approaches were based on taking the intersection of the French and English lexicons of the Treetagger Part of Speech tagger (Schmid, 1994), after applying a set of lexical transforma- tions. The submission was a “quick and dirty hack” implemented on little sleep during confer- ence attendance, and merely demonstrates that a simple heuristic-based algorithm can beat a man- ually curated list of cognates, albeit not by much. However, the approach should perform better than demonstrated in the ALTW challenge if applied more comprehensively. 3 Algorithms The first “quick and dirty” baseline algorithm (Al- gorithm 1) only looks for exact matches once case and accents are preprocessed: 1. Replace the accented letters in the French lexicon with unaccented letters. For example, replace “ˆe” and “´e” with “e”. 2. Casefold and remove punctuation from the words in the source and cognate file. 3. Take the intersection of the sanitised source and cognate file words. 4. All words in the text that are in the intersec- tion are counted as cognates. Algorithm 2 discards any Algorithm 1 words of 5 letters or less as false friends. Common false friend English words that become discarded as a result include: “a”, “as”, “an”. However, the fol- lowing cognates are also discarded: “ah”, “oh”. Algorithm 3 uses lexical transformations to the French lexicon list before intersecting with the En- glish …

Content truncated to 500 words