prompt-programs

This paper studies performance prediction for symbolic (e.g., Python) and prompt programs using a Bayesian coin-flip model, finding that symbolic programs have all-or-nothing performance while prompt programs have diffuse priors, and introduces RAP (Retrieved Approximate Prior) for performance prediction.