Teaching a humanoid robot to play cornhole with Gemini 4 Argon

**TL;DR: A DeepMind experiment enabled Gemini 4 to learn to control a humanoid robot playing cornhole by watching human videos, practicing in a simulator, and iteratively correcting its mistakes, ultimately landing all seven bags in the hole consecutively.** # Teaching a Humanoid Robot to Play Cornhole: The Gemini 4 Experiment ## Experimental Goal and Overall Approach In this Google DeepMind experiment, the research team explored whether Gemini 4 could be taught to control a humanoid robot to play cornhole. The overall process was straightforward: - First, show Gemini 4 a video of a human performing the task; - Then prompt Gemini 4 to run the Mojoku simulator to practice; - Next, let it learn from its own experience, repeatedly iterate, make mistakes, and review video recordings of each failed attempt; - With the experience accumulated from each attempt, it could continually improve and perform better. Throughout the process, no human programmed the movements frame by frame. Instead, the model was allowed to master this physical skill through observation, simulation, and self-correction. ## Starting with a Perfect Throw One of the most striking moments in the experiment was the robot throwing a bag that landed directly in the hole: > Wow! An airmail straight into the hole. Awesome! According to the robot’s own description, it used four steps to throw the bag accurately into the hole. ## Step One: Learning Movements from a Human Video First, it watched a video of a person throwing a bag, tracked the person’s arm swing and release, and then mapped those human movements to its own robotic joints and hands. In other words, video understanding served as the starting point for "motion transfer"—extracting executable joint-level movements from human body movements. ## Step Two: Building a 3D Physical Scene Second, the robot used its two eye cameras to measure the target board’s exact three-dimensional position and built a physics simulation containing both itself and the board. The two eye cameras provided depth perception, allowing the robot to reconstruct the board’s real-world position in the simulated environment and lay the groundwork for extensive trial-and-error practice. ## Step Three: Practicing Throws in the Simulator Third, it practiced throwing in the simulated environment, making specific adjustments that included: - **Widening its stance**; - **Adjusting its arm swing** so its hand could clear its hip cleanly; - **Throwing in a smooth arc**. This step placed all trial and error inside the simulator, where the robot could try repeatedly without worrying about wear on real hardware or safety issues. ## Step Four: Transferring to the Real Robot and Fine-Tuning Fourth, it returned to the real robot. It observed where each bag actually landed and adjusted the release position by a few centimeters each time, until it ultimately **threw all seven bags directly into the hole consecutively**. The transfer from simulation to reality was not completed in a single step. Instead, it relied on small adjustments based on feedback from where each bag landed. ## Key Capabilities: Video Reasoning and Learning from Experience What made the experiment truly exciting was the two capabilities demonstrated by Gemini 4: 1. **Reasoning through video understanding**, allowing it to narrow down exactly what went wrong; 2. **Learning from its own experience**, turning each failure into a basis for improving the next attempt. As stated in the video: > Gemini 4 can reason through its video understanding to narrow down exactly what went wrong, and it can learn from its own experience, which just seems incredibly exciting. ## Summary This experiment demonstrated a complete path to acquiring a skill, from "watching human demonstrations" to "autonomous practice in simulation" and then "real-world fine-tuning." The robot did not need rigid movement rules programmed one by one. Instead, through video understanding, physics simulation, and iterative learning from experience, it independently refined a rough imitation into a consistent performance of seven consecutive successful throws. Source: Teaching a humanoid robot to play cornhole with Gemini 4 Argon — Google DeepMind (https://www.youtube.com/watch?v=9PICEmtfpWU)