Five AI Systems, Same Prompts, Twice: Wildly Different Responses

Running identical prompts through five different AI systems twice reveals substantial variation in outputs, according to testing conducted by Thomas D. Holt and published in Towards AI in June 2026. The experiment highlights a critical challenge for developers building production applications: language models do not guarantee consistent responses even when given identical inputs.

TL;DR

Holt’s experiment tested five AI systems with the same prompts submitted twice to each model. The results showed wildly different responses across systems and even between runs on the same system. This variability stems from how language models generate text probabilistically rather than deterministically. For developers relying on AI outputs in production workflows, this inconsistency presents both technical and reliability challenges that require specific mitigation strategies.

Detailed Breakdown

Language models sample from probability distributions when generating each token, meaning identical prompts can produce different outputs. While some systems offer temperature settings or seed parameters to control randomness, complete reproducibility remains difficult to achieve across different models and API versions.

The five-system comparison revealed variation along several dimensions. Response length differed significantly between runs, even when prompts requested specific formats. Tone and style shifted unpredictably, with some outputs adopting formal language while others used conversational phrasing for the same query. Factual content also varied, particularly for questions requiring reasoning or synthesis rather than simple retrieval.

This variability compounds when comparing different AI systems. Each model architecture, training dataset, and fine-tuning approach produces distinct response patterns. What works reliably on one system may fail or produce unexpected results on another, making cross-platform consistency nearly impossible without additional engineering.

Code Examples

Developers can attempt to reduce variability by setting explicit parameters:

 model="gpt-4",
 messages=[{"role": "user", "content": "Explain quantum computing"}],
 temperature=0.0,
 seed=42
)

Setting temperature=0.0 reduces randomness by making the model select the most probable token at each step. The seed parameter provides deterministic sampling in some implementations, though support varies by provider.

For critical applications requiring consistent outputs, developers should implement validation layers:

 if not matches_schema(response):
 return retry_with_adjusted_prompt()
 return response

Gotchas

The biggest pitfall is assuming AI outputs will remain stable over time. Model updates, infrastructure changes, and even API load can affect response characteristics. Applications built around specific output formats may break when providers update their systems.

Temperature settings do not eliminate variation entirely. Even at zero temperature, different model versions or API endpoints may produce different results. The seed parameter, where available, only ensures reproducibility within the same model version and infrastructure configuration.

Prompt engineering alone cannot guarantee consistency across systems. A carefully crafted prompt that produces reliable results on one model may yield unpredictable outputs on another. Developers should test prompts across target systems and implement fallback logic for unexpected response formats.

Caching responses based on prompt text seems logical but can create problems. Since identical prompts may produce different outputs, cached responses might not represent the current model’s behavior. This becomes particularly problematic when models are updated or when applications switch between providers.

For production systems, the recommendation is to treat AI outputs as probabilistic rather than deterministic. Build validation, retry logic, and human review workflows around the assumption that responses will vary. Document which specific model versions and settings produce acceptable results, and monitor for drift when those configurations change.