Less is More: Summary of Long Instructions is Better for Program Synthesis

Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 4532 - 4552 December 7-11, 2022 ©2022 Association for Computational Linguistics Less is More: Summary of Long Instructions is Better for Program Synthesis Kirby Kuznia∗ Swaroop Mishra∗ Mihir Parmar Chitta Baral Arizona State University Abstract Despite the success of large pre-trained lan- guage models (LMs) such as Codex, they show below-par performance on the larger and more complicated programming related questions. We show that LMs benefit from the summa- rized version of complicated questions. Our findings show that superfluous information of- ten present in problem description such as hu- man characters, background stories, and names (which are included to help humans in under- standing a task) does not help models in under- standing a task. To this extent, we create a meta- dataset from the frequently used APPS dataset and the newly created CodeContests dataset for the program synthesis task. Our meta-dataset consists of human and synthesized summaries of the long and complicated programming ques- tions. Experimental results on Codex show that our proposed approach outperforms baseline by 8.13% on the APPS dataset and 11.88% on the CodeContests dataset on average in terms of strict accuracy. Our analysis shows that sum- maries significantly improve performance for introductory (9.86%) and interview (11.48%) programming questions. However, it shows improvement by a small margin (∼ 2%) for competitive programming questions, implying scope for future research in this direction.1 1 Introduction Recently, large pre-trained LMs have been proven pivotal in programming-related tasks (Wang et al., 2021; Chen et al., 2021; Hendrycks et al., 2021; Lu et al., 2021; Papineni et al., 2002)2. Program synthesis aims to generate a code given the natural language description of a problem. Programming requirements in these problems vary in terms of complexity from a 3-5 line simple function to mul- tiple functions that use advanced data structures. ∗Equal Contribution 1Code and data is available at https://github.com/kurbster/ Prompt-Summarization 2Detailed related work is presented in Appendix A However, LMs such as Codex show below-par per- formance on the long and complicated program- ming questions. We observe that the natural lan- guage description of the program becomes long and complicated when there is superfluous infor- mation (see section 2.1.1). The goal of adding this information to the description is to make it more understandable to humans. However, we find that this information confuses the model in understand- ing a task3. We propose that removing the excess information and providing the model with the ex- act specifications of the problem can improve the performance of the LMs. To remove excess information4, we summarize the descriptions of the program in such a way that it does not lose important specifications. We use the APPS dataset (Hendrycks et al., 2021) and Code- Contests dataset (Li et al., 2022) which are a col- lection of coding problems from different online sources and create a meta-dataset consisting of hu- man and synthesized summaries. We perform all experiments using the GPT-based Codex model (Chen et …

Content truncated to 500 words