Claude Plays RollerCoaster Tycoon via CLI

Game automation typically requires complex computer vision pipelines or direct memory manipulation. A developer recently demonstrated a simpler approach: letting Claude “see” game screenshots and issue keyboard commands through a command-line interface, effectively turning the AI into a game-playing agent without writing game-specific code.

Bridging AI Vision and Legacy Games

The project addresses a fundamental challenge in AI-game interaction. Classic games like RollerCoaster Tycoon weren’t designed with API access or programmatic control. Traditional automation requires reverse-engineering the game’s memory structure or training specialized models on thousands of gameplay hours. This CLI-based approach sidesteps both requirements by treating the game as a visual interface that Claude can observe and control through standard input methods.

The system captures screenshots at regular intervals, sends them to Claude’s vision API, and translates Claude’s natural language responses into actual keyboard and mouse commands. When Claude responds “click on the path tool in the bottom left,” the CLI parser converts this into precise coordinates and executes the click. This creates a feedback loop where Claude observes outcomes, adjusts strategy, and issues new commands.

Architecture and Command Flow

The implementation relies on three core components working in sequence. A screenshot utility captures the game window at configurable intervals, typically every 2-3 seconds to balance responsiveness with API costs. These images get encoded and sent to Claude via the Anthropic API along with context about previous actions and current objectives.

Claude analyzes each frame and returns structured commands. The parser recognizes patterns like “click(x, y)”, “press(key)”, or “wait(seconds)” and routes them to the appropriate system libraries. On Windows, this might use pyautogui or win32api; on Linux, xdotool handles input simulation.

import anthropic
from PIL import ImageGrab
import pyautogui

client = anthropic.Anthropic(api_key="your-key")

def capture_game():
    screenshot = ImageGrab.grab()
    screenshot.save("frame.png")
    return screenshot

def get_claude_action(image_path, history):
    with open(image_path, "rb") as img:
        response = client.messages.create(
            model="claude-3-5-sonnet-20241022",
            max_tokens=1024,
            messages=[{
                "role": "user",
                "content": [
                    {"type": "image", "source": {"type": "base64", "data": img}},
                    {"type": "text", "text": "What action should we take? Respond with click(x,y) or press(key)."}
                ]
            }]
        )
    return parse_command(response.content)

The conversation history maintains context across frames, allowing Claude to remember it just built a roller coaster entrance or set admission prices. This memory proves essential for multi-step tasks like constructing complex ride layouts or managing park finances over time.

Building Your Own Game Agent

Setting up the system requires Python 3.8+ and several dependencies. Install the Anthropic SDK, Pillow for image handling, and an input automation library appropriate for your operating system. The repository at https://github.com/anthropics/anthropic-quickstarts contains reference implementations.

Configuration involves defining the game window boundaries to ensure consistent screenshot regions. Most implementations use a calibration step where you manually mark the game area, storing coordinates for subsequent captures. Setting appropriate API rate limits prevents excessive costs during long play sessions.

The prompt engineering significantly impacts performance. Effective prompts include the game’s objective, available UI elements, and constraints like budget limits. Providing Claude with game-specific knowledge—“guests prefer gentle rides when park rating is low”—improves decision quality without requiring training data.

Testing should start with simple objectives: build one food stall, construct a basic path network, or hire a single mechanic. These bounded tasks help debug the command parser and verify Claude correctly interprets UI elements before attempting complex park management.

Integration with Broader AI Tooling

This approach fits into the emerging ecosystem of vision-language model applications. Similar techniques enable AI agents to navigate desktop applications, complete web forms, or control simulation environments. The pattern generalizes: capture visual state, query a vision model, execute parsed commands, repeat.

The project demonstrates practical applications for Claude’s computer use capabilities, which allow the model to interact with standard software interfaces. While Anthropic’s official Computer Use API provides more robust tooling, this CLI implementation shows how developers can build custom agents for specific applications.

Performance limitations include API latency (typically 1-3 seconds per decision) and cost considerations for extended sessions. Future iterations might implement local caching of common UI states or hybrid approaches where Claude handles strategic decisions while rule-based systems manage repetitive tasks like path maintenance.