Great study on what actually predicts vibe-coding ability.

100 tertiary-level students completed measures of computer-science achievement, domain-general cognitive skills and written-communication proficiency, then took a vibe-coding assessment.

The study was preregistered, which matters because the intuitive answer here, that writing ability carries everything, is what a post-hoc analysis would tend to find.

Tasks were curated through an eight-expert consensus process and run in a purpose-built vibe-coding environment that mirrors commercial tools while allowing controlled measurement.

They find that writing skill predicts performance. So does computer-science achievement, and it stays significant after controlling for domain-general cognitive skills, so it is measuring something specific to CS training rather than general ability.

The practical question the authors point at is a curriculum one. How much time to spend teaching people to write good prompts, against how much to spend on computer-science fundamentals, for the people who will build software this way.

Paper: https://academy.dair.ai/papers/computer-science-achievement-and-writing-skills-predict-vibe-coding-proficiency-2603.14133…

Computer Science Achievement and Writing Skills Predict Vibe Coding Proficiency

Source: https://academy.dair.ai/papers/computer-science-achievement-and-writing-skills-predict-vibe-coding-proficiency-2603.14133 EvaluationChat with Paper

First page

The curator’s take

Sverrir Thorgeirsson, Theo B. Weidmann and Zhendong Su at ETH Zurich run a preregistered cross-sectional study of 100 tertiary-level students to find out which measured skills predict how well someone performs at vibe coding.

Ask this paper

Question about this paper Key points01

Preregistered design. 100 students completed measures of computer-science achievement, domain-general cognitive skills, and written-communication proficiency, plus a vibe-coding assessment. Preregistration matters here because the obvious result, that writing skill alone carries it, is exactly the kind of claim a post-hoc analysis would produce.

02

Tasks were built for controlled evaluation. An eight-expert consensus process curated the tasks, which ran in a purpose-built vibe-coding environment that mirrors commercial tools while allowing controlled measurement.

03

Both predictors are significant. Writing skill and CS achievement each significantly predict vibe-coding performance, so natural-language ability alone does not account for the outcome.

04

CS achievement survives the obvious control. It remains a significant predictor after controlling for domain-general cognitive skills, which rules out the reading that it is a proxy for general intelligence.

05

Where it applies. The authors frame the result for tool and curriculum design, specifically the question of when to emphasize prompt writing against CS fundamentals.

AbstractMany software development platforms now support LLM-driven programming, or “vibe coding”, a technique that allows one to specify programs in natural language and iterate from observed behavior, all without directly editing source code. While its adoption is accelerating, little is known about which skills best predict success in this workflow. We report a preregistered cross-sectional study with tertiary-level students (N = 100) who completed measures of computer-science achievement, domain-general cognitive skills, written-communication proficiency, and a vibe-coding assessment. Tasks were curated via an eight-expert consensus process and executed in a purpose-built, vibe-coding environment that mirrors commercial tools while enabling controlled evaluation. We find that both writing skill and CS achievement are significant predictors of vibe-coding performance, and that CS achievement remains a significant predictor after controlling for domain-general cognitive skills. The results may inform tool and curriculum design, including when to emphasize prompt-writing versus CS fundamentals to support future software creators.