Controlling Android with Gemini 3.7 Flash and 150 lines of Python

Last week we launched Gemini 3.7 Flash, our first hybrid reasoning model built for coding and agentic workflows.

Its multimodal reasoning and fast turn-by-turn latency make it uniquely suited for Computer Use. Instead of waiting several seconds between taps, the model inspects screenshots, plans the next step, and emits precise coordinates fast enough to keep device control loops interactive.

To test this on a real mobile interface, I gave it a task: “play 1 round of Wordle at wordle.global/en/unlimited”.

Running on an Android emulator, it opened Chrome, cleared tutorial popups, played an opening word, read the color-coded feedback with thinking, and solved the game in 2 guesses.

Running on an Android emulator, it opened Chrome, cleared tutorial popups, played an opening word, read the color-coded feedback with thinking, and solved the game in 2 guesses.

The problem with mobile automation

Mobile test automation has been painful for 15 years. Appium, Espresso, and UIAutomator all depend on accessibility trees, element IDs, and XPath selectors. When a product team runs an A/B test, redesigns a checkout card, or adds a marketing banner, the test script breaks.

Webviews and dynamic canvas games are even worse because they often expose zero accessibility nodes to ADB.

Gemini 3.7 Flash allows you to approach this almost human like. The model looks at the raw screenshot, figures out where elements sit, and returns normalized 0–999 coordinates for clicks, text input, and key presses.

How the agent loop works

The Agent connects to an Android emulator over ADB and runs a continuous loop:

• Capture: ADB takes a screenshot (adb exec-out screencap -p) and encodes it as Base64.

• Evaluate: Gemini 3.7 Flash inspects the image and plans its next move.

• Emit action: Gemini returns a tool call with normalized coordinates and intent (e.g. click(x=884, y=190)).

• Execute: The Python script maps 0–999 to physical pixels (1080x1920) and fires adb shell input tap .

• Continue: The client grabs a fresh screenshot and sends it back in function_result with previous_interaction_id.

The full implementation, including the ADBBridge coordinate scaler, device setup scripts, and CLI runner, is in the gemini-android-computer-use-quickstart repository.

Example: Playing 1 round of Wordle

Here is what happened during the Wordle session.

1. Launch and navigation

The agent started on the home screen, opened Chrome by package name, opened a new tab, and typed the URL:

2. Clearing modal overlays

Two popups blocked the game board on page load: a “How to Play” tutorial and a stats dialog. The model spotted the close icons and clicked them:

3. Playing guess 1 (CRANE)

With a clear board, it typed a standard vowel-heavy opener: C - R - A - N - E.

4. Reading tile colors with thinking

The game returned feedback:

• C turned yellow (in the word, wrong spot).

• R, A, N turned gray (not in the word).

• E turned green (correct spot at index 5).

Gemini checked words ending in E that contain C somewhere in the middle without R, A, or N. It picked SPICE:

All 5 tiles turned green. The model saw the victory popup and stopped:

What you can build with this

Wordle is a clean visual benchmark, but the underlying loop works for any Android application.

You can use this for automated UI testing, user flow verification, exploratory bug reproduction, or general task automation. Because the model operates on raw screenshots, it works across native Android apps, webviews, and dynamic canvas interfaces without relying on accessibility IDs or DOM trees.

The complete quickstart with the emulator setup script, ADB bridge, and CLI runner is open source:

• GitHub repo: google-gemini/gemini-android-computer-use-quickstart

• Interactions API overview: Interactions API Docs

• Computer Use guide: Computer Use Guide

• Thinking documentation: Thinking in Gemini 3.7