Gemini Plays Pokemon - About (Archived June 12, 2025)
Background
Hi, I'm a 30 year old software engineer unaffiliated with Google. Welcome! :)
With the release of Gemini 2.5 Pro Experimental, there was much interest in testing how well it plays Pokémon.
After a successful first run, we're now kicking off our second—fully uninterrupted—on May 17 to test all the improvements made along the way. The current model version is Gemini 2.5 Pro Preview 05-06.
On May 22, ClaudePlaysPokemon just relaunched with Claude 4, so by popular request we’ve restarted too—now you can watch both AIs side-by-side and compare their very different harnesses. This is all in good fun so don't treat this as a serious competition!
How It Works
This project is an autonomous AI player for Pokémon Blue. Here's how it works:
Core Components
- Game Interface: The code connects to an mGBA emulator running Pokémon Blue via a socket connection, allowing it to:
- Take screenshots of the current game state
- Send button press commands (A, B, Up, Down, etc.)
- Retrieve game status data like player position, Pokémon party info, and map details
- Decision Making: The system processes game screenshots with a grid overlay and sends them (along with some game state extracted from RAM) to Google's Gemini 2.5 Pro, which analyzes the information given and decides what to do next. In some situations, it may defer to a specialized agent for improved reasoning and reliability.
The Game Loop
The main game loop follows this sequence:
- Capture a screenshot and retrieve game status data
- Process the image with a grid overlay to help with spatial reasoning
- Send the screenshot and game information to the AI model
- The AI decides whether to respond directly or call a specialized agent
- Parse the response to determine which button to press
- Execute the button press and wait for the game to update
- Repeat the process for the next frame
Memory Management
Messages are currently summarized roughly every 100 actions to reduce input tokens used. The summary replaces those messages. This number is subject to change as the experiment progresses.
Agents
Pathfinder – Another instance of the LLM that helps Gemini find its way through complex areas like spinner mazes. It plans a path using reasoning alone, through simulating algorithms like BFS (though it can choose whichever method it thinks is best).
Boulder Puzzle Strategist (BPS) – Also an LLM instance, added for the boulder puzzles in Victory Road. The agent mentally simulates sequences of boulder pushes to determine valid solutions. While Gemini could occasionally succeed without the agent, it focuses Gemini's reasoning capabilities to handle these challenges with increased reliability.
FAQ
Q: What's different in the second run?
A: The first "second run" launched on May 17 (PDT). On May 22 we reset the run so it could start in lock-step with ClaudePlaysPokemon's Claude 4 relaunch. The fresh run is identical to the aborted one (with minor harness improvements), but now viewers can watch Gemini and Claude side-by-side and compare how each LLM tackles the exact same game from the same starting point, exploring each model and their harnesses' strengths and weaknesses. This is purely for fun, so don't treat this as a serious race! Follow both streams together here (multi-view).
With the reset, Gemini starts completely fresh—its memory is wiped, and the run begins from scratch. There are no changes to prompts or tooling, so this run serves as a clean test of all the improvements made during the first run. There won't be any developer interventions unless Gemini becomes hard-stuck due to a system limitation. That said, it may be a matter of weeks before a situation is considered truly hard-stuck. In that case, any necessary improvements will be made and the run will be reset.
Q: What is GG? What is the “Walkthrough” that GG gives?
A: GG stands for “Guidance Gemini,” an observer agent that critiques Gemini’s performance every 25 turns based on its last 100 actions. As part of this critique, GG provides a “Walkthrough” suggesting what to do next. This guidance is generated from Gemini’s internal training data—it is not pulled from any external source or official guide, and can often be incorrect or incomplete. An external walkthrough was considered early on, but ultimately ruled out to keep things fair.
Q: Why Pokémon Blue?
A: ClaudePlaysPokemon is already streaming Pokémon Red, and Blue was the first Pokémon game I ever played, so it was a natural choice!
Q: Why does the game look different?
A: I'm using \"Danny-E 33's Gen 2 Graphics Patch,\" which updates the visuals to Generation 2's style.
Q: How does Gemini's navigation work?
A: Gemini doesn't have an automatic navigation tool like Claude. Instead, it independently chooses its destinations and can either plot a manual path to get there, or invoke a pathfinding agent (another instance of Gemini) to find the best path.
Q: What's the minimap-like thing above the game?
A: It's a visual representation of Gemini's current exploration. The map gradually fills in as Gemini explores new areas, akin to fog-of-war in other video games. A textual representation of this is also included in Gemini's context, but not the image itself.
Q: Which is better, Claude or Gemini?
A: Please don't consider this a benchmark for how well an LLM can play Pokémon. You can't really make direct comparisons—Gemini and Claude have different tools and receive different information. Each model thinks differently and excels in unique ways. However, Claude's framework has many shortcomings, so I wanted to see how far Gemini could get if it were given the right tools. Watch both and decide for yourself which you prefer!
Q: I've heard you frequently help Gemini (dev interventions, etc.). Isn't this cheating?
A: No, it's not cheating. During the first run, Gemini Plays Pokémon was still in active development, and I made occasional interventions to improve its general decision-making and reasoning abilities. These were never specific hints—there were no walkthroughs or direct instructions for areas like Mt. Moon. The only exception was letting Gemini know that it needed to talk to a Rocket Grunt twice to get the Lift Key, a bug that was later fixed in Yellow. Claude Plays Pokémon underwent similar behind-the-scenes refinements before streaming began. Learn more about that here.
Now, in the second run, Gemini is fully autonomous. See "What's different in the second run?" above.
Q: Isn't giving Gemini a minimap cheating?
A: That's subjective, but I personally don't consider it cheating. Humans naturally build mental maps while playing games, something current LLMs can't do independently yet. The fog-of-war minimap feature compensates for that limitation.
Roadmap (tentative)
- improved memory management?
- allow Gemini to write down notes for important stuff?
- give Gemini more information about the game state (done)
- allow chat to interact with Gemini in some way while not allowing hints through (done in first run's post-game, removed for 2nd run, and will be back in a future run)
- Completely uninterrupted run once scaffolding is "complete" (ongoing)
- Runs with a different LLM for comparison, e.g. Claude or o3