Data Collection vs. Knowledge Graph Completion: What is Needed to Improve Coverage?

Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 6210–6215 November 7–11, 2021. c⃝2021 Association for Computational Linguistics 6210 Data Collection vs. Knowledge Graph Completion: What is Needed to Improve Coverage? Kenneth Church Baidu USA kennethchurch@baidu.com Yuchen Bian Baidu USA yuchenbian@baidu.com Abstract This survey/position paper discusses ways to improve coverage of resources such as Word- Net. Rapp estimated correlations, ρ, between corpus statistics and psycholinguistic norms. ρ improves with quantity (corpus size) and qual- ity (balance). 1M words are enough for simple estimates (unigram frequencies), but at least 100M are required for pairs of words (word as- sociations, edges). Knowledge Graph Comple- tion (KGC) attempts to learn missing links in WN18. Unfortunately, WN18 is flawed with information leaking from train to test. More se- riously, WN18 is based on SemCor (just 200k words) and dated (collected in 1960s). KGC cannot learn anything that happened since the 1960s, or associations requiring 100M words. 1 Quantity (Size) and Quality (Balance) How large do the corpora have to be to learn what? In the early 1980s, corpora were about 1M words. The Brown Corpus (Kuˇcera and Francis, 1967; Francis and Kuˇcera, 1979, 1982) was large enough for first order statistics (counts of words), but not for second order statistics (word associations and counts of pairs of words). The Brown Corpus was a balanced corpus. That is, the corpus was intended to be a representative sample of text that the system will see at inference time. The 1M word Brown Corpus consists of 500 samples1 of 2000 words, representative of contem- porary American English (from 1960s). Over time, balanced corpora became larger. When the community decided to increase the size of balanced corpora from 1M words for the Brown Corpus to 100M for the British National Corpus (BNC) (Aston and Burnard, 1998; Burnard, 2002), 1500 samples span 15 categories: Press Reportage (44 texts), Press Editorial (27), Press Reviews (17), Religion (17), Skills and Hobbies (36), Popular Lore (48), Miscellaneous US Government & House Organs (30), Learned (80), General Fic- tion (29), Mystery and Detective Fiction (24), Science Fiction (6), Adventure and Western (29), Romance (29), Humor (9). it was known that 1M was too small for second or- der statistics (collocations and word associations), but it was hoped that 100M would be sufficient. Around this time, Church and Hanks (1990) used an unbalanced sample of 44M words from the AP (Associated Press) to make the case for PMI (point- wise mutual information). Given the estimates in Table 1, it appears in retrospect that 44M words were just barely enough to make the case for PMI. It was also believed that quality (balance) mat- tered, but there were few, if any, empirical studies to justify such beliefs. It was extremely contro- versial when engineers such as Mercer questioned these deeply held beliefs in 19852 with: “there is no data like more data.” Most people working on corpus-based methods in lexicography were deeply committed to balance as a matter of faith, and …

Content truncated to 500 words