Shorter Is Not Cheaper: A Controlled Comparison of Declarative Animation Specifications and Free-Form Code for LLM-Generated Explanations

arXiv cs.CL Papers
llm-explanations declarative-animation reasoning-models evaluation-study arxiv benchmarking

Summary

研究通过预注册的受控实验比较了声明式动画规格与自由HTML/JavaScript代码两种LLM生成解释的方案，发现规格虽更短，但由于推理开销增加并未更便宜、更快或更可靠，小模型的规格可靠性甚至显著更低。

arXiv:2610.32129v1 Announce Type: new Abstract: Chat assistants increasingly answer conceptual questions with interactive, animated explanations, generated either as free-form HTML/JavaScript or as declarative specifications that a fixed runtime plays. The declarative route is widely expected to be cheaper, faster and more reliable, because its outputs are shorter and constrained. We test this expectation in controlled comparisons whose hypotheses and analysis plans were frozen before the runs. We designed ExplainSpec, a time-indexed declarative animation format, and an observation contract that makes every generated explanation programmatically inspectable; validated, pacing-agnostic checkers score 20 explanation requests from five domains. Three Claude models (360 jobs) and, in a follow-up, the open-weight Qwen3-14B with thinking enabled and disabled (240 jobs) answered every request in both formats. For every model the specifications were 25-48% shorter in answer tokens (output excluding reasoning), yet with reasoning enabled total output fell significantly only for the largest Claude model and no model generated significantly faster. The reasons differed. The Claude models reasoned far more before writing specifications (exploratory; for the two larger models, from 15-20% to 85-88% of generations). Qwen3-14B reasoned about as much in both formats (a pre-registered test of this shift was not supported), and its shorter answers saved little against a reasoning trace more than twice as long. Nor were specifications more reliable: free-form pages were valid after one repair round in 95-100% of jobs for every model, whereas Qwen3-14B produced valid specifications in only 33% of jobs and passed all semantic tests in 17% (HTML/JavaScript: 62%). For reasoning models, a shorter format alone therefore does not make generation cheaper, faster or more reliable, and a format learned in context can cost reliability.
Original Article

Similar Articles

Best patterns for evaluating AI agents?

Reddit r/AI_Agents

A practitioner shares their deterministic evaluation harness for QA/coding agents — fixed corpora, injected bugs, hidden tests, and pass^k/token metrics — and asks the community how others balance smoke tests vs full benchmarks, whether LLM-as-a-judge adds value, and whether tools like Promptfoo or DeepEval reduce maintenance burden.

Could a modern chess engine on a Pentium have defeated Kasparov in 1996?

Hacker News Top

Nicholas Carlini demonstrates that a modern Stockfish port running on a 1996-era 200 MHz Pentium Pro achieves ~3300 Elo, far surpassing Kasparov's ~2800, showing that most of the +1335 Elo gain since 1996 comes from better programming (+720) rather than faster hardware — meaning Deep Blue's custom supercomputer was never necessary.