Training on Lexical Resources

Proceedings of the 13th Conference on Language Resources and Evaluation (LREC 2022), pages 6290–6299 Marseille, 20-25 June 2022 © European Language Resources Association (ELRA), licensed under CC-BY-NC-4.0 6290 Training on Lexical Resources Kenneth Church, Xingyu Cai, Yuchen Bian Baidu, Sunnyvale, USA {kennethchurch,xingyucai,yuchenbian}baidu.com Abstract We propose using lexical resources (thesaurus, VAD) to fine-tune pretrained deep nets such as BERT and ERNIE. Then at inference time, these nets can be used to distinguish synonyms from antonyms, as well as VAD distances. The inference method can be applied to words as well as texts such as multiword expressions (MWEs), out of vocabulary words (OOVs), morphological variants and more. Code and data are posted on https://github.com/kwchurch/syn_ant. Keywords: Synonyms, Antonyms, Fallows, VAD (Valenace, Arousal, Dominance), multiword expressions 1. Introduction: Training on Lexicons We do not normally think of lexical resources as training data, though others have trained on dic- tionaries (Brown et al., 1993; Chairatanakul et al., 2021). Suppose we have a thesaurus such as (Fal- lows, 1898).1 Consider the thesaurus to be a set of triples: w1, w2, rel, where w1 and w2 are two words, and rel is 0 if w1 and w2 are synonyms and 1 if they are antonyms. We can then fine-tune a pretrained deep net such as BERT (Devlin et al., 2019) or ERNIE (Sun et al., 2020) using eq (1). rel ∼ w1 + w2 (1) The fine-tuning code is very simple. We modified an example from HuggingFace2 in straightforward ways.3 This code takes a pretrained net as input, and a set of triples, and outputs a fine-tuned net. text1 text2 y1 y2 good bad -3.95 4.54 bad evil 4.44 -5.00 good benevolent 4.43 -5.05 bad benevolent -3.44 4.16 good terrorist -3.43 4.10 bad terrorist 4.48 -5.10 Table 1: Inference: synonymy iff y1 > y2 Then there is another program for inference.4 The in- ference program takes a fine-tuned net, and a pair of texts, and outputs two logits, y1 and y2, as illustrated in Table 1. The program predicts that the two input texts are synonymous iff y1 > y2. 1https://www.gutenberg.org/files/ 51155/51155-0.txt 2https://github.com/huggingface/ accelerate/blob/main/examples/nlp_ example.py 3https://github.com/kwchurch/syn_ant/ blob/main/sentiment4.py 4https://github.com/kwchurch/syn_ant/ blob/main/sentiment4_inference.py In Table 1, the first two columns, text1 and text2, are single words, but the inference program accepts arbi- trary texts as input (up to 512 subword units). Table 2 shows inference on multiword expressions (MWEs)5 (Baldwin and Kim, 2010). It has been said that multi- word expressions (MWEs) are a pain in the neck for natural language engineering (Sag et al., 2002), though MWEs are less of a pain in the neck for the proposed approach than for alternatives. In other words, the inference program uses eq (2) to predict y from two input texts: y ∼ text1 + text2 (2) This notation is inspired by general linear models in R6 (Guisan et al., 2002). We will start with binary classi- fication (logistic regression). Later, classification will be replaced with regression when we consider VAD (Valance, Arousal and Dominance) distances in §5. text1 text2 y1 y2 freedom …

Content truncated to 500 words