Adapting to the Long Tail: A Meta-Analysis of Transfer Learning Research for Language Understanding Tasks

Adapting to the Long Tail: A Meta-Analysis of Transfer Learning Research for Language Understanding Tasks Aakanksha Naik1,3 Jill Lehman2 Carolyn Ros´e1,3 1Language Technologies Institute, Carnegie Mellon University, USA 2Human-Computer Interaction Institute, Carnegie Mellon University, USA 3Rehabilitation Medicine Department, Clinical Center, National Institutes of Health, USA {anaik,jfl,cp3a}@andrew.cmu.edu Abstract Natural language understanding (NLU) has made massive progress driven by large bench- marks, but benchmarks often leave a long tail of infrequent phenomena underrepresented. We reflect on the question: Have transfer learning methods sufficiently addressed the poor performance of benchmark-trained mod- els on the long tail? We conceptualize the long tail using macro-level dimensions (underrep- resented genres, topics, etc.), and perform a qualitative meta-analysis of 100 representative papers on transfer learning research for NLU. Our analysis asks three questions: (i) Which long tail dimensions do transfer learning stud- ies target? (ii) Which properties of adaptation methods help improve performance on the long tail? (iii) Which methodological gaps have greatest negative impact on long tail perfor- mance? Our answers highlight major avenues for future research in transfer learning for the long tail. Lastly, using our meta-analysis framework, we perform a case study com- paring the performance of various adaptation methods on clinical narratives, which provides interesting insights that may enable us to make progress along these future avenues. 1 Introduction ‘‘There is a growing consensus that significant, rapid progress can be made in both text under- standing and spoken language understanding by investigating those phenomena that occur most centrally in naturally occurring unconstrained materials and by attempting to automatically ex- tract information about language from very large corpora.’’ (Marcus et al., 1993) Since the creation of the Penn Treebank, using shared benchmarks to measure and drive progress in model development has been instrumental for accumulation of knowledge in the field of natural language processing, and has become a dominant practice. Ideally, we would like shared benchmark corpora to be diverse and comprehensive, which can be addressed at two levels: (i) macro-level dimensions such as language, genre, topic, and so forth, and (ii) micro-level dimensions such as spe- cific language phenomena like negation, deixis, causal reasoning, and so on. However, diversity and comprehensiveness are not straightforward to achieve. According to Zipf’s law, many micro-level lan- guage phenomena naturally occur infrequently and will be relegated to the long tail, except in cases of intentional over-sampling. Moreover, the advantages of restricting community focus to a specific set of benchmark corpora and lim- itations in resources lead to portions of the macro-level space being under-explored, which can further cause certain micro-level phenom- ena to be under-represented. For example, since most popular coreference benchmarks focus on English narratives, they do not contain many in- stances of zero anaphora, a phenomenon quite common in other languages (e.g., Japanese, Chi- nese). In such situations, model performance on benchmark corpora may not be truly reflective of expected performance on micro-level long tail phenomena, raising questions about the ability of state-of-the-art models to generalize to the long tail. Most benchmarks do not explicitly …

Content truncated to 500 words