DEEP_DIVE_STORY.LNK

SWE-Bench-style coding scores are spiking, but contamination and self-reported leaderboards mean you should trust results only after your own verification.

Zhipu’s GLM-4.7 touts a 200K context window, low token costs, and a 73.8% SWE-Bench score, positioning it as a coding workhorse with strong math and instruction-following claims GLM-4.7. These specs look great for large monorepos and long diffs.

Meanwhile, the SWE-Bench Pro leaderboard is dominated by proprietary models, but every score is self-reported and shows 0 verified results SWE-Bench Pro. A separate head-to-head guide shows GPT-5.4 winning terminal workflows while Claude Opus 4.6 edges web search synthesis, underscoring that “best” depends on task mix and tier GPT‑5.4 vs Claude Opus 4.6.

New research argues coding benchmarks face a credibility crisis and offers Cross-Context Verification to detect contamination: run N isolated sessions and score solution diversity; lack of real reasoning is a tell Cross-Context Verification. They report perfect separation on tested cases and a chunk of prior labels as false positives.