NEW SWE-BENCH PROMAX RAISES THE BAR FOR AI AGENTS WITH HARD, MULTILINGUAL REFACTORING TASKS
SWE-Bench ProMax debuts a tougher, multilingual refactoring benchmark that fixes test flaws and exposes where AI coding agents still break. ProMax is a curated...
SWE-Bench ProMax debuts a tougher, multilingual refactoring benchmark that fixes test flaws and exposes where AI coding agents still break.
ProMax is a curated set of 170 real refactoring tasks across seven languages, with rewritten specs and cleaned tests to avoid known issues in earlier suites. Tasks average 11.4 files and 261 lines changed, and the best reported agent only resolves 41.2% — a clear gap versus prior benchmarks paper.
This lands alongside data showing verification isn’t keeping up with AI-generated code — 80% of orgs traced incidents to AI code and testing often lags (report, interview). Separate tracks like SWE-bench Multimodal are evolving in parallel but measure different skills leaderboard.
Your internal agent evals based on older suites likely overestimate success on real refactors.
Verification and test authoring need to scale with generation or production risk grows.
-
terminal
Run a ProMax-style trial: pick a 10+ file refactor, rewrite a precise spec, scrub flaky tests, and measure agent resolve rate vs human baseline.
-
terminal
Split generation and verification: use a different model or human reviewers to author tests and validate diffs; compare to self-graded results.
Legacy codebase integration strategies...
- 01.
Gate agent-generated refactors behind protected branches with repo-specific linters, CONTRIBUTING rules, and full CI matrices before merge.
- 02.
Tag provenance for AI-generated commits to speed incident tracing and postmortems.
Fresh architecture paradigms...
- 01.
Design for agent-friendly refactors: explicit specs, high test coverage, stable fixtures, and deterministic builds.
- 02.
Bake independent verification into the pipeline (separate model/human QA) and track pass rates on cross-file changes.
Get daily THE-NEW-STACK + SDLC updates.
- Practical tactics you can ship tomorrow
- Tooling, workflows, and architecture notes
- One short email each weekday