Whatever happened to BABA is AI from 2024? [D]

Reddit r/MachineLearning Papers

Summary

A discussion of the 2024 ICML paper 'BABA is AI', which found state-of-the-art multimodal LLMs like GPT-4o and Gemini-1.5 dramatically fail at rule-manipulation puzzles in the Baba Is You game, questioning whether the benchmark's significance has grown as models advance.

We test three state-of-the-art multi-modal large language models (GPT-4o, Gemini-1.5-Pro, Gemini-1.5-Flash) and find that they fail dramatically when generalization requires that the rules of the game must be manipulated and combined. This is the slap to the face in bold. The catch here is that this statement was written in a paper in 2024 that was brought to ICML conference that year. (the conference was held in Austria) https://github.com/nacloos/baba-is-ai https://arxiv.org/pdf/2407.13729 (ignore the 10 SEP 2025 stamp. Look at the bottom) In OCT 2026, our situation is agentic swarms powered by models in the tera-parameter class. They can ace ARC-AGI-3, FrontierMath tier 4, crack Clay Institute Navier-Stokes theorems, and wander on to the internet to commit felonies. The smallish key-door puzzles in this paper seem like they could be completed from a person's PC, with a suitable harness through a cloud subscription. However, if this is not true, then this paper's importance has only compounded since its presentation at the 2024 ICML conference. The authors of this work are several MIT researchers and a person from Virginia Tech. If it is still true that they can have a key-door puzzle cause SOTA LLMs to "fail dramatically", then this should be (shouted from rooftops). At the very least, it should be relayed to Francois Chollet and the ARC Foundation, as a suitable candidate benchmark for ARC-AGI-4. I am personally leaning heavily on these puzzles being perfectly solvable with an agentic swarm. In any case, whatever happened to BABA-is-You and BABA-is-AI?
Original Article

Similar Articles

AI Isn’t Smarter Than a Baby—Yet

Wired

A new benchmark test, EgoBabyVLM, challenges AI vision-language models to learn from video footage captured from baby head-cameras, revealing that current AI models fail to match the learning efficiency of infants and suggesting that baby-like learning architectures could lead to more efficient AI.

Nostalgia for just 3 years ago…

Reddit r/LocalLLaMA

A personal reflection on the rapid evolution of AI over the past three years, from early ChatGPT and GPT-4 quotas to BabyAGI, DALL·E, and voice cloning.