Tag
Flagsmith 推出 Experiments 功能，基于已有 feature flags 运行开源 A/B 测试，使用贝叶斯方法判断胜出变体，并支持托管仓库或自带 ClickHouse/Databricks。
A practitioner reports 1,680 paired A/B trials showing that classic Unix and CS formalisms (Circuit Breaker FSM, SysV init-d runlevels, git bisect, Design-by-Contract, 2PC+WAL, EBNF) dramatically outperform verbose English prompt rules for LLM agent reliability, arguing that prompting is essentially static symbol linkage on high-frequency training-corpus constructs.
Yandex Music introduces Sona, a single transformer that replaced 15+ candidate generators plus pre-ranker and ranker in production A/B tests, using History Compression to halve inference cost while delivering +4.53% Active Users and +6.30% Listening Time.
This paper investigates how historical A/B test data can inform adaptive experiments using contextual bandits, providing a practical methodology for deciding when and how to deploy adaptive policies based on offline policy evaluation.
The article outlines nine prompt discipline rules that reduced wasted thinking in coding agents by up to 70% based on A/B tests on GLM 5.3 and GLM 5.3 Flash models, with a self-testable exam available on GitHub.
The paper introduces Odds-Ratio Thompson Sampling (OR-TS), a method for batched multi-armed bandits that uses contrast-based updates to handle varying common levels, showing improved regret over absolute-rate memory in simulations and real experiments.
Introduces Tree-Coupled A/B Testing (TCAB), an exact feedback-sharing design for comparing multiple adaptive policies with fewer reward queries while preserving each policy's trajectory law.
Microsoft Research highlights new research on SocialRL for small language model negotiation, PazaBench V2 for African language speech evaluation, EvoLib for agent experience learning, improved A/B testing methods, and AI-driven precision oncology.
A software engineer discusses when feature flags make sense, such as for A/B testing and complex deployments, and cautions against overusing them when teams have control over their deployments.
GrowthBook 5.0 is a new release of the open-source feature flag and A/B testing platform, focused on helping teams build, ship, and improve at scale.
Corbin Braun built an evolutionary A/B thumbnail testing tool that mutates one dimension at a time, rotates thumbnails using an ABBA pattern to avoid bias, and learns winning mutations over multiple rounds.
Amboras is an AI-native ecommerce platform that automates store optimization, including A/B testing and layout changes, aiming to boost conversion rates significantly.
Spotify Engineering discusses using LLM evals as a funnel before A/B experiments, improving hit rates and creating a feedback loop between evals and experiments.
The author introduces Syrin, a runtime A/B testing tool for AI agents that allows teams to run controlled experiments on live traffic across prompts, models, and agent topologies. They are seeking 5-10 engineering teams to test the tool in production and provide feedback.