Whatever happened to BABA is AI from 2024? [D]
Summary
A discussion of the 2024 ICML paper 'BABA is AI', which found state-of-the-art multimodal LLMs like GPT-4o and Gemini-1.5 dramatically fail at rule-manipulation puzzles in the Baba Is You game, questioning whether the benchmark's significance has grown as models advance.
Similar Articles
AI Isn’t Smarter Than a Baby—Yet
A new benchmark test, EgoBabyVLM, challenges AI vision-language models to learn from video footage captured from baby head-cameras, revealing that current AI models fail to match the learning efficiency of infants and suggesting that baby-like learning architectures could lead to more efficient AI.
AI models got way more reliable but the AI labs forgot to make them smarter (SciCode)
The article argues that AI models have become more reliable but intelligence gains have plateaued, suggesting older models like GPT-4 turbo could rival newer ones with proper enhancements.
AI can finally pass the Turing Test better than a human, study warns
A new study published in PNAS shows that advanced LLMs like GPT-4.5 can pass the Turing Test, with participants finding them more human than actual humans, prompting a reevaluation of what the test measures.
Nostalgia for just 3 years ago…
A personal reflection on the rapid evolution of AI over the past three years, from early ChatGPT and GPT-4 quotas to BabyAGI, DALL·E, and voice cloning.
@steverab: Very excited to share that our paper "Towards a Science of AI Agent Reliability" was accepted at ICML 2026! See you in …
A paper analyzing AI agent reliability, accepted at ICML 2026, finds that even the latest frontier models (GPT 5.5, Gemini 3.1 Pro, Claude Opus 4.7) show only marginal reliability improvements over earlier versions, with low outcome consistency and persistent issues in agent scaffolding.