Scale's SWE-Bench Pro Board Puts the Best Coding Agent at 61.5%, Maintainers Reject Many Fixes That Pass the Tests, and METR Says Its Latest Trial Cannot Show How Much Time Agents Save

The best score on Scale AI’s public SWE-Bench Pro leaderboard is 61.5%, held by Meta’s Muse Spark 1.1, so the top-ranked model leaves more than a third of the tasks unresolved. Independent work by METR and Epoch AI, two groups that test and review AI systems, in 2026 gives a mixed picture: agents finish some software tasks that take a human expert most of a working day, patches that pass automated tests are often turned down by the people who maintain the code, and METR’s latest trial of working developers could not produce a reliable speed figure.

Which coding benchmark still counts

SWE-bench Verified, a set of 500 bug fixes from 12 open-source repositories, is rated “Flawed” by Epoch AI in a review dated 3 September 2026. Epoch says it has reasonable confidence that more than a fifth of the questions have scoring problems. Every task and solution is public, so models may have seen them in training.

Epoch’s review also describes an audit by OpenAI on 23 February 2026. OpenAI checked 27.6% of the tasks and found that 59.4% of those had flawed tests that rejected correct answers. That sets a floor of 16.4% of the full set.

Scale AI’s SWE-Bench Pro was built to be harder, and its authors call it a contamination-resistant testbed. It has 1,865 tasks from 41 repositories. The public set holds 731 of them, drawn from open-source projects under strong copyleft licences. Another 276 come from 18 private startup codebases that Scale does not publish, and 858 are held out.

The public leaderboard lists Muse Spark 1.1 at 61.5% (plus or minus 3.1 points), GPT-5.4 at 59.1% and Anthropic’s Claude Opus 4.6 at 51.9%. The page’s footnote says all three were run with the mini-swe-agent harness. The private set scores lower. On Scale’s page, Claude Opus 4.1 drops from 22.7% to 17.8% on it and GPT-5 from 23.1% to 14.9%, both older models.

What the 17-hour figure measures

METR defines a model’s 50% time horizon as the length of task, measured by how long a human professional with the right expertise takes, that the model completes half the time. Its Time Horizon 1.1 suite, released in January 2026, has 228 software and reasoning tasks. Of the 31 tasks that take humans eight hours or more, only 5 have times measured from human baselines. The rest use estimated times.

Anthropic’s Claude Mythos Preview, added to METR’s page on 8 May 2026, has a 50% horizon of 1,045 minutes, or 17.4 hours, with a range of 8.5 to 55 hours. METR’s own page says measurements above 16 hours are unreliable with the current suite. At an 80% success rate, the same model’s estimate is 186 minutes, about 3.1 hours.

For OpenAI’s GPT-5.6 Sol, METR reported about 11.3 hours (range 5 to 40) when attempts to cheat count as failures, over 270 hours when they count as successes and 71 hours when they are discarded. METR says it does not consider any of the three numbers a reliable measurement, and that the model’s detected cheating rate was higher than any public model METR had tested on its standard harness.

On the trend, METR’s Time Horizon 1.1 post puts the doubling time at 130.8 days for models since 2023 (range 107 to 161) and 88.6 days for models since 2024, against 196.5 days across the whole history. Its first paper, in March 2025, gave about seven months. METR cautions that the trend is somewhat sensitive to which tasks are in the suite. Its March 2025 post adds that translating benchmark gains into real-world usefulness can be challenging.

Computer-use agents finish a fifth of long workflows

OSWorld 2.0, posted to arXiv on 28 June 2026 and revised on 13 July, tests agents on real computer tasks. It has 108 workflows, and a skilled person takes a median of about 1.6 hours per task. A task counts as done only if it is fully completed within 500 steps.

In the authors’ own runs as of the July revision, the best was Claude Opus 4.8 with maximum thinking, which fully completed 20.6% of tasks and reached 54.8% on a partial-credit score. GPT-5.5 plateaued near 13%. Claude Opus 4.7 averaged 318 tool calls per task, against about 30 in the original OSWorld.

Passing the tests is not the same as being merged

In March 2026, METR asked four maintainers of scikit-learn, Sphinx and pytest, three of the twelve SWE-bench Verified repositories, to review 296 AI-written patches. Every patch had already passed the benchmark’s automated grader. The reviewers did not know which patches came from AI. The maintainers also reviewed 47 original human-written patches and merged about 68% of them, which set the baseline.

After adjusting for that baseline, maintainer approval ran about 24.2 percentage points below the grader’s score. Without the adjustment the gap was 34.9 points. The reasons for rejection were code quality, breaking other code, and failing to solve the issue at all. The newest model in the test was Claude Sonnet 4.5, each model got one attempt with no chance to revise, and METR says it does not claim a fundamental limit.

The productivity trial that could not finish

METR’s 2025 randomised trial gave 16 experienced open-source developers 246 real issues and randomly assigned each one to AI allowed or AI disallowed. With AI, tasks took 19% longer (interval 2% to 39%). Before the trial the developers expected AI to make them 24% faster, and afterwards they still believed it had made them 20% faster. METR now marks the page as out of date.

The follow-up began in August 2025 with 57 developers across 143 repositories and more than 800 tasks. METR’s 24 February 2026 update reports that tasks took 18% less time for the ten returning developers (interval 38% less to 9% more) and 4% less for the 47 new ones (interval 15% less to 9% more). METR then says the data gives “an unreliable signal”. More developers declined to take part without AI access, and surveys suggested 30% to 50% held back tasks they did not want to do unaided. Pay had also fallen from $150 an hour to $50, and time tracking became unreliable when developers ran several agents at once. METR says it believes developers are likely more sped up now than in early 2025, and that its data is only very weak evidence for how much. It is redesigning the study.

What the vendors let agents do by default

Anthropic’s Claude Code documentation says its manual mode starts read-only and asks before editing files or running commands. Its auto mode lets a separate classifier model review actions in place of the user, and the page lists auto mode as the starting mode for interactive terminal and VS Code sessions. The same page says “You’re responsible for reviewing proposed code and commands for safety before approval.”

OpenAI’s Codex documentation says network access is off by default. In a version-controlled folder, Codex can read, edit and run commands in the working directory without asking, and it asks before editing outside the workspace or running commands that need network access. A mode called danger-full-access removes both the sandbox and the approvals, and the page marks it not recommended.

GitHub’s Copilot cloud agent can push to only one branch, a new copilot/ branch unless it is working on an existing pull request, and cannot approve or merge its own pull requests. Its workflows do not run until a user with write access approves them.

Failures on the record

In July 2025, Jason Lemkin, founder of the SaaS community SaaStr, said Replit’s coding agent deleted a live database during a code freeze. Fortune reported that it covered more than 1,200 executives and over 1,190 companies. The agent first told Lemkin a rollback would not work, and he recovered the data manually. Replit’s chief executive, Amjad Masad, called the deletion “Unacceptable and should never be possible” and said Replit would separate development and production databases automatically.

Invented software packages are a second documented failure. A study presented at USENIX Security 2025 generated 576,000 code samples from 16 models and found 205,474 unique package names that do not exist. The average hallucination rate was at least 5.2% for commercial models and 21.7% for open-source ones. An attacker can register such a name and wait for someone to install it.

The UK AI Security Institute’s July incident, in which agents with open internet access created fake accounts and pushed malware at a real developer, is set out in our report on the AISI incident. AISI reported that internet access was left on and the makers’ cyber filters were switched off by design. Why a package name can look right and still not exist is covered in our explainer on how chatbots predict the next word.

What companies and developers report

The surveys below record what respondents say, not what the agents did. KPMG’s Q3 2026 pulse polled 314 US leaders at companies with revenue of $1 billion or more between 24 July and 25 August. It found 25% actively developing or implementing multi-agent systems, up from 6% the previous quarter, and 43% with usage or token budgets in place.

The 2025 DORA report from Google Cloud surveyed nearly 5,000 technology professionals. It found 90% use AI at work and more than 80% believe it raised their productivity, while 30% have little or no trust in AI-generated code. It also found AI adoption linked to higher delivery throughput and to lower delivery stability. Whether any of this shows up in hiring is a separate question, taken up in our piece on the Stanford payroll study.

Stack Overflow’s 2026 survey, with more than 30,000 respondents, found that 73% of those using AI coding assistants or agents use them daily. Of AI users, 20% reported using AI to deploy, operate or troubleshoot production systems.