Do CoNLL-2003 Named Entity Taggers Still Work Well in 2023?
38365Authors
nan
Abstract
AbstractThe CoNLL-2003 English named entity recognition (NER) dataset has been widely used to train and evaluate NER models for almost 20 years. However, it is unclear how well models that are trained on this 20-year-old data and developed over a period of decades using the same test set will perform when applied on modern data. In this paper, we evaluate the generalization of over 20 different models trained on CoNLL-2003, and show that NER models have very different generalization. Surprisingly, we find no evidence of performance degradation in pre-trained Transformers, such as RoBERTa and T5, even when fine-tuned using decades-old data. We investigate why some models generalize well to new data while others do not, and attempt to disentangle the effects of temporal drift and overfitting due to test reuse. Our analysis suggests that most deterioration is due to temporal mismatch between the pre-training corpora and the downstream test sets. We found that four factors are important for good generalization: model architecture, number of parameters, time period of the pre-training corpus, in addition to the amount of fine-tuning data. We suggest current evaluation methods have, in some sense, underestimated progress on NER over the past 20 years, as NER models have not only improved on the original CoNLL-2003 test set, but improved even more on modern data. Our datasets can be found at https://github.com/ShuhengL/acl2023_conllpp.
Keywords
Full Content
Self-Edit: Fault-Aware Code Editor for Code Generation Kechi Zhang, Zhuo Li, Jia Li ♂, Ge Li∗, Zhi Jin∗ Key Lab of High Confidence Software Technology (PKU), Ministry of Education School of Computer Science, Peking University, China {zhangkechi,lizhmq}@pku.edu.cn, lijia@stu.pku.edu.cn, {lige,zhijin}@pku.edu.cn Abstract Large language models (LLMs) have demon- strated an impressive ability to generate codes on competitive programming tasks. However, with limited sample numbers, LLMs still suffer from poor accuracy. Inspired by the process of human programming, we propose a generate- and-edit approach named Self-Edit that utilizes execution results of the generated code from LLMs to improve the code quality on the com- petitive programming task. We execute the gen- erated code on the example test case provided in the question and wrap execution results into a supplementary comment. Utilizing this com- ment as guidance, our fault-aware code editor is employed to correct errors in the generated code. We perform extensive evaluations across two competitive programming datasets with nine different LLMs. Compared to directly gen- erating from LLMs, our approach can improve the average of pass@1 by 89% on APPS-dev, 31% on APPS-test, and 48% on HumanEval over nine popular code generation LLMs with parameter sizes ranging from 110M to 175B. Compared to other post-processing methods, our method demonstrates superior accuracy and efficiency. 1 Introduction Large language models (LLMs) have recently been applied to the competitive programming task. This task requires understanding a complex natural lan- guage description of a problem with example test cases and correctly implementing solutions that can span hundreds of lines. Solutions are evalu- ated by executing them on hidden test cases. How- ever, existing LLMs often have low accuracy and pass rates in this task. For example, on a popular competitive programming benchmark APPS-test (Hendrycks et al., 2021), the nearly most powerful model GPT3 (Brown et al., 2020) achieves only 7% accuracy when allowed to submit only one program per task (referred to as pass@1). *Corresponding authors write an initial code execute on example test cases refine and debug submit generate an initial code from LLM execute on example test cases refine with a neural-based editor submit Human Programmers Our approach (a) GPT3 Output: n, k = map(int, input().split()) if n % k == 0: print n + k else: print n + k - (n % k) Error Message: SyntaxError: Missing parentheses in call to 'print'. Did you mean print(n + k)? Problem Description: APPS-test-673 Find the smallest integer x greater than n, so it is divisible by the number k. The input only line contains two integers n and k (1 ≤ n, k ≤ 10^9). -----Examples----- Input 5 3 Output 6 example test case (b) Figure 1: (a) Our approach is inspired by the problem- solving process of human programmers. (b) Output from GPT3 model on APPS-test dataset and its corre- sponding error message, which is obtained by running on the example test case. To improve the performance of LLMs on the competitive programming task, we take inspiration from the process of human programming. …
Content truncated to 500 words
Labeled Text
AbstractThe CoNLL-2003 English named entity recognition (NER) dataset has been widely used to train and evaluate NER models for almost 20 years. However, it is unclear how well models that are trained on this 20-year-old data and developed over a period of decades using the same test set will perform when applied on modern data. In this paper, we evaluate the generalization of over 20 different models trained on CoNLL-2003, and show that NER models have very different generalization. Surprisingly, we find no evidence of performance degradation in pre-trained Transformers, such as RoBERTa and T5, even when fine-tuned using decades-old data. We investigate why some models generalize well to new data while others do not, and attempt to disentangle the effects of temporal drift and overfitting due to test reuse. Our analysis suggests that most deterioration is due to temporal mismatch between the pre-training corpora and the downstream test sets. We found that four factors are important for good generalization: model architecture, number of parameters, time period of the pre-training corpus, in addition to the amount of fine-tuning data. We suggest current evaluation methods have, in some sense, underestimated progress on NER over the past 20 years, as NER models have not only improved on the original CoNLL-2003 test set, but improved even more on modern data. Our datasets can be found at
Publication Details
- Paper ID:
- 38365
- Year:
- 2023
- Publication:
- TIPSTER
- Publication Type:
- conf
- DOI:
- nan