Vibe vs Claude Code: Near-Identical SWE-Bench Results

What It Is

A recent benchmark study put Mistral’s Vibe (powered by Devstral 2) head-to-head with Anthropic’s Claude Code across 900 test runs on SWE-bench-verified-mini, a dataset containing 45 real-world GitHub issues. The methodology involved running each AI coding agent multiple times on the same problems to measure both success rates and consistency.

The headline numbers landed remarkably close: Claude Code (using Opus through auto-selection) solved 39.8% of issues, while Vibe achieved 37.6%. That 2.2 percentage point gap falls within statistical noise given the sample size. More striking than the similarity in solve rates was what emerged about reproducibility - roughly 40% of test cases produced inconsistent results across multiple runs with the same agent. On problems that agents solved successfully in all 10 attempts, the generated patches varied in size by up to 8x between runs.

The full analysis with detailed charts and methodology is available at https://blog.kvit.app/posts/variance-claude-vibe/

Why It Matters

This comparison challenges two common assumptions about AI coding tools. First, the performance parity between an open-weight model and a proprietary frontier system suggests the gap between open and closed models has narrowed considerably for code generation tasks. Vibe delivered comparable results while completing runs faster (296 seconds average versus 357 seconds for Claude Code), making it particularly relevant for teams with budget constraints or data residency requirements.

Second, and perhaps more significant, the variance findings expose a fundamental measurement problem in AI coding benchmarks. When 40% of problems yield different outcomes across identical runs, single-execution benchmarks become unreliable indicators of real-world performance. A model might solve a problem on one attempt and fail on the next, or generate a minimal fix in one run and a sprawling refactor in another.

This inconsistency has practical implications for development workflows. Teams integrating AI coding agents cannot treat them as deterministic tools that produce the same output given the same input. The variance means developers need strategies for handling unpredictable behavior - perhaps running agents multiple times on critical fixes or implementing review processes that account for solution instability.

Getting Started

Developers interested in testing Vibe can access it through Mistral’s API or run Devstral 2 locally since it’s open-weight. The model is available through standard inference frameworks:

client = MistralClient(api_key="your_api_key")
response = client.chat(
 model="devstral-2",
 messages=[{"role": "user", "content": "Fix the bug in this code..."}]
)

For Claude Code, developers can access it through Anthropic’s API at https://console.anthropic.com or through integrated development environments that support Claude integration.

The benchmark methodology itself offers lessons for evaluation. Rather than relying on single runs, the study executed 20 attempts per issue (10 with each agent). This approach reveals not just whether an agent can solve a problem, but how reliably it does so. Teams evaluating coding agents for production use should consider similar multi-run testing on their specific codebases.

Context

Traditional code generation benchmarks like HumanEval and MBPP measure performance on isolated programming challenges with clear correct answers. SWE-bench differs by using real GitHub issues that require understanding existing codebases, navigating project structure, and generating patches that pass actual test suites. This makes it more representative of production coding tasks but also introduces the variance observed in this study.

The inconsistency likely stems from temperature settings and sampling strategies in the underlying language models. Most coding agents use non-zero temperature to encourage diverse solutions, which introduces randomness into token selection. While this can help agents explore different approaches, it also means identical inputs don’t guarantee identical outputs.

Alternative approaches exist for teams prioritizing consistency over creativity. Deterministic code completion tools like GitHub Copilot in certain modes or rule-based static analysis tools produce reproducible results, though they may lack the flexibility of agent-based systems. The choice depends on whether a workflow values solution diversity or predictable behavior.