Tag
This paper presents DNSSE, a hybrid framework combining LLM-guided schedule prediction, symbolic constraint solving, and coverage-guided stochastic mutation for verifying distributed parallel AI programs. It detects 2.9x more concurrency bugs than the baseline and raises branch coverage from 68.6% to 91.6% on realistic benchmarks.
MetaSpace is a framework that applies metamorphic testing to evaluate spatial cognition in embodied agents, generating test cases from execution trajectories and encoding logical/physical constraints as Prolog rules. Benchmarking shows state-of-the-art MLLM-driven agents score far below human levels on spatial cognition, with directional tasks being especially weak.
A field note argues that a green test suite doesn't prove tests are meaningful; AI agents should be asked to make tests fail to verify they actually catch bugs.
freeCodeCamp announces a comprehensive course on software testing with Playwright, covering fundamentals like the testing pyramid, TDD, API testing, and AI-powered tools like KaneAI.
A blog post by Antithesis detailing bugs found in several open-source Raft consensus implementations, including HashiCorp Raft and OpenRaft, emphasizing the difficulty of testing distributed systems and the need for better tooling.
Antithesis describes its custom deterministic hypervisor and subtree deduplication technique for efficient, repeatable whole-system fuzz testing at cloud scale.
A detailed blog post from Antithesis describing how their software testing tool was used to explore and achieve 'rebirth' in Tetris by reaching level 255, overcoming challenges and discovering glitches along the way.
Momentic announces a major platform update with an AI-powered knowledge base and autonomous testing agents to address the growing gap between code velocity and software quality.
A tweet argues that AI app testing should be a first-class feature in coding apps, noting that many obvious problems could be caught if AI tried the app itself.
Using Codex to automate app testing by generating user stories and tracking feature status in a spreadsheet through iterative loops.
Gergely Orosz shares his experience using Antithesis, a deterministic testing infrastructure that can run hours of testing in minutes.
This article argues that standard test suites with fixed inputs and expected outputs are insufficient for AI agents due to infinite input spaces and non-deterministic behavior, advocating for property-based testing instead.
The article discusses using LLMs as automated QA engineers to perform manual testing tasks, such as integration and regression testing, potentially raising software quality bar.
The author shares a TDD skill for AI agents to improve test writing, based on Kent Beck's Canon TDD, and provides a GitHub link.
The author tested Persistent Sage's long-term memory feature, finding it accurately recalled personal facts like colorblindness and a spouse's name from a week earlier without explicit prompting, demonstrating effective persistent memory for AI agents.
After leaving the workplace, the author is curious whether the workflow of QA in big companies remains the same—submitting a ticket after finding a bug—and believes that submitting a bug can itself be seen as a prompt for AI, so it might be better to directly let AI modify the code.
This article explores why adding assumptions to specifications or tests inherently weakens the resulting property, using logical implication and examples from formal methods and Rust. It also discusses practical reasons for using assumptions despite this weakening.
A tweet shares a prompt that configures Composer 2.5 to act as a QA engineer, creating test documentation and bug reports for development phases.
A developer connected Claude Code to an iPhone simulator and prompted it to 'test everything,' resulting in autonomous navigation, bug detection, and structured reporting, showcasing a shift from traditional test scripts to AI-driven intent-based testing.
This post implements a Daikon-style runtime invariant miner in Python, including instrumentation, trace collection, candidate invariant checking, and implication-based suppression, providing an approximate oracle for regression testing.