Isola: cloud LLMs could soon puppeteer connected robots

"This year we are going to see many LLMs being tested as robot-use agents," Phillip Isola opens in a [short essay dated September 7](https://web.mit.edu/phillipi/www/writing/robot-use-agents.html) on his MIT page. Two of the researchers we track shared the link within a day. The framing is Claude-as-puppeteer: a language model driving a robot body the way it now drives a browser or a calculator. Isola grants the standing objections about latency, weak spatial reasoning, and thin physical intuition, but writes that "these limitations are starting to melt away with newer models like Fable and Astra." The consequence he draws is a distribution shift more than a capability jump. Robot intelligence today lives on-device, tuned to each hardware line, with researchers still splitting bets across world models, behavior foundation models, and continual learning. Cloud puppeteering skips all of that. "Every robot with an internet connection becomes a potential tool for Fable, Astra, and other AIs," Isola writes; "a car might be as easily controlled as a factory arm." He does not claim any of this is better yet. LLM-controlled robots, he notes, "are still far less performant than dedicated solutions," and puppeteering "may be less reliable than tried-and-true robotic systems; this matters especially for high-stakes and safety-critical use cases." The essay names no specific demos and cites no benchmarks. It is a positioning argument about where the robotics stack may be about to fracture, not a claim about what current systems can do today.