AGENTS ACE ONE-SHOT CODING, BUT MOST BREAK YOUR CODE OVER MONTHS—TIME TO HARDEN CI AND ADOPT EVALUATOR LOOPS
New results say most coding agents cause regressions during long-term CI, and a new MassGen release adds built-in evaluator loops to catch issues earlier. A ne...
New results say most coding agents cause regressions during long-term CI, and a new MassGen release adds built-in evaluator loops to catch issues earlier.
A new write-up of the SWE-CI benchmark shows that one-shot fixes don’t predict sustained quality: most models regress in over 75% of tasks across months of repo evolution, with only Claude Opus clearing a 50% zero-regression rate TLDR Dev 2026-03-09. For context, one-shot ability like SWE-Bench Verified measures “coding IQ,” not maintenance stamina SWE Bench Verified - AI Benchmark.
The bottleneck isn’t more tests, it’s stable execution. Deterministic, isolated, production-faithful environments and signals that converge on correctness are prerequisites for agentic QA at scale Notable Capital. On the tooling side, MassGen v0.1.61 ships a “round evaluator” subagent that auto-spawns parallel evaluators after each answer, feeding critiques into the next round and offering a ready-made loop for catching mistakes before merge MassGen v0.1.61.
If you worry agents only handle popular stacks, recent experience suggests good harnesses and local docs overcome that bias—agents can learn your unfamiliar tools when guided and evaluated well Simon Willison.
Agent-generated code can look correct in PRs yet quietly increase long-term regression risk without stronger evaluation and deterministic CI.
Tools are emerging to close the gap; adopting evaluator loops and hermetic environments can turn flashy demos into reliable delivery.
-
terminal
Run a SWE-CI-style replay on your repos: reapply historical PRs with your agent harness and measure zero-regression rate across multi-week CI loops.
-
terminal
Trial MassGen’s round_evaluator example on a non-critical service and compare pre-merge defect rates and time-to-green versus your current agent workflow.
Legacy codebase integration strategies...
- 01.
Gate agent-authored PRs behind hermetic CI: ephemeral environments, frozen dependencies, seeded data, and API record/replay to cut flakes.
- 02.
Add regression sentinels (golden tests, schema diff checks, latency/error budgets) and require evaluator-loop pass before human review.
Fresh architecture paradigms...
- 01.
Design for determinism from day one: idempotent migrations, hermetic builds, reproducible datasets, and isolated service sandboxes.
- 02.
Pick an agent harness that supports multi-agent evaluation and documented Skills, and budget for continuous evaluation telemetry.