Tag
A research paper examining non-determinism and event overcount in modern hardware performance counter implementations, analyzing why performance measurements may not be reproducible across runs and platforms.
A reflective blog post arguing that non-determinism in LLM output is not a fatal flaw for software development, since only the end result needs to be exact—not the path taken. The author advocates using AI to build deterministic, tested tools that agents can call, thereby harnessing randomness today to reduce randomness in future systems.
This paper identifies the hidden injection of the current date into LLM system prompts as an overlooked source of evaluation non-determinism, showing performance deltas of up to 14% on math reasoning and up to 6% on MCQA across 9 recent LLMs, with effects exceeding batch size and precision variations and even shifting model rankings on leaderboards.
The article expands Gary Bernhardt's 'Functional Core, Imperative Shell' architecture to 'Deterministic Core, Non-Deterministic Shell,' highlighting determinism over pure functionalism for better testability and broader applicability in software systems.
This article critically examines the accuracy of AI visibility tools that claim to measure brand presence in generative AI responses, arguing that they provide false precision due to nondeterminism, personalization, and scraping biases. It calls for transparency in methodology and warns against treating opaque dashboards as stable truth.
A discussion on the challenges of testing AI agent harnesses with non-deterministic components, exploring approaches like golden output diffing and using an LLM as a judge, while questioning the validity of such methods.
This article argues that standard test suites with fixed inputs and expected outputs are insufficient for AI agents due to infinite input spaces and non-deterministic behavior, advocating for property-based testing instead.