SWE-rebench: A Decontaminated SWE Agent Benchmark

SWE-rebench is an automated pipeline and public dataset for collecting software engineering tasks and evaluating the agents that try to solve them. It was introduced in a paper titled “SWE-rebench: An Automated Pipeline for Task Collection and Decontaminated Evaluation of Software Engineering Agents,” published at NeurIPS 2025. The work comes from a team of researchers including Ibragim Badertdinov, Alexander Golubev, Boris Yangel, and several colleagues, and the dataset is hosted under the Nebius organization on Hugging Face. The paper is available at https://arxiv.org/abs/2505.20411.

How the Pipeline Collects Tasks

The central idea behind SWE-rebench is automation. Rather than assembling a fixed set of problems by hand, the pipeline continuously extracts real-world, interactive software engineering tasks from a range of GitHub repositories. The tasks are Python-based and require an agent to interact with a development environment, execute code, and adapt its approach based on the results it observes.

The authors describe the pipeline as novel, automated, and scalable. Because it keeps pulling new tasks from active repositories, the supply of problems does not stay frozen at a single point in time. That ongoing collection produced a dataset of more than 21,000 interactive Python tasks, which the paper notes is large enough to support reinforcement learning of software engineering agents at scale.

Why Contamination Matters

A recurring problem with static coding benchmarks is contamination. When a fixed test set has been public for a while, parts of it can end up in the training data of newer language models. Once that happens, a model may appear to perform well not because it reasons through the problem but because it has effectively seen the answer before. Over time, this makes older benchmarks less trustworthy as a measure of genuine capability.

SWE-rebench addresses this by using its fresh, continuously collected tasks to build what the authors call a contamination-free benchmark. The reasoning is straightforward: tasks drawn from recent repository activity are far less likely to have appeared in a model’s training data.

What the Evaluation Revealed

To test their concern directly, the authors compared how language models scored on SWE-rebench against their results on SWE-bench Verified, an existing and widely cited benchmark. The comparison indicated that the measured performance of some language models might be inflated due to contamination issues. In other words, scores on the older benchmark may overstate how well certain models actually handle unfamiliar software engineering problems.

That finding is the practical payoff of the decontamination approach. By continually refreshing its task pool, SWE-rebench aims to give a cleaner read on how agents perform on problems they have not encountered before.

Where to Find It

The dataset is published on Hugging Face under the Nebius organization, and a public leaderboard is available at swe-rebench.com for tracking how different systems perform. Together with the paper, these resources let researchers inspect the tasks, reproduce evaluations, and compare agents on a benchmark that is designed to stay current rather than grow stale.